tag:blogger.com,1999:blog-66699345955999378942024-03-12T23:23:51.878-07:00Obvious Hints"null is null or not an object" IE6 WisdomAlvarohttp://www.blogger.com/profile/05177930414107959806noreply@blogger.comBlogger37125tag:blogger.com,1999:blog-6669934595599937894.post-58429883892613544582011-04-15T04:27:00.000-07:002011-05-17T06:47:51.257-07:00It's so quiet here why?I forgot to mention that I've moved my blog here: <a href="http://videlalvaro.github.com/">http://videlalvaro.github.com/</a><br /><br />I'm also writing a book about RabbitMQ, check it out here: <a href="http://bit.ly/rabbitmq">http://bit.ly/rabbitmq</a><br /><br />Thanks for watching!Alvarohttp://www.blogger.com/profile/05177930414107959806noreply@blogger.com0tag:blogger.com,1999:blog-6669934595599937894.post-77280316584140185382010-07-13T00:05:00.000-07:002010-07-13T00:15:40.451-07:00A note about OOP reusabilityI was reading the first chapter of the book <a href="http://www.amazon.com/Practical-Clojure-Experts-Voice-Source/dp/1430272317/">Practical Clojure</a>, when I found this paragraph comparing OOP code vs. Functional Programming code:<br /><br /><cite>"It [OOP] encourages a high degree of ceremony and code bloat. Simple functionality in Java can require several interdependent classes. Efforts to reduce close coupling through techniques like dependency injection involve even more unnecessary interfaces, configuration files, and code generation. Most of the bulk of a program is not actual program code, but defining elaborate structures to support it."</cite><br /><br />Which takes me back to my thoughts about all the verbosity of OOP code vs. say Haskell code. If we need so many things to make OOP code work or produce something useful, then are we sure this is the right programing paradigm?Alvarohttp://www.blogger.com/profile/05177930414107959806noreply@blogger.com5tag:blogger.com,1999:blog-6669934595599937894.post-56877755739193188622010-05-18T05:48:00.000-07:002010-05-18T05:54:07.422-07:00About Haskell Static TypingFrom time to time I re-read some random chapter of <a href="http://www.realworldhaskell.org/">Real World Haskell</a>. Last night I picked chapter two and found this interesting paragraph about static typing:<br /><br /><cite>"A helpful analogy to understand the value of static typing is to look at it as putting pieces into a jigsaw puzzle. In Haskell, if a piece has the wrong shape, it simply won't fit. In a dynamically typed language, all the pieces are 1x1 squares and always fit, so you have to constantly examine the resulting picture and check (through testing) whether it's correct."</cite><br /><br />This pops in my mind all the <span style="font-weight: bold;">is_a*</span> tests that I've seen in PHP code.Alvarohttp://www.blogger.com/profile/05177930414107959806noreply@blogger.com1tag:blogger.com,1999:blog-6669934595599937894.post-72190803209033817132010-04-21T03:48:00.000-07:002010-04-21T03:56:17.500-07:00A Word About Caching: Memcached and APCSometimes when talking with developers I see that there are some misconceptions regarding this two caching systems. That's why I'd like to share some concepts I've learned along the way.<br /><br />What does APC do?<br /><br />APC has two main features, one is to cache the PHP opcodes, to speed up the page delivery. This means that our PHP code doesn't have to be parsed every time it gets executed. The other one is to cache user variables, so you can cache the results of <span style="font-weight: bold;">expensive_function()</span> in APC, using <span style="font-weight: bold;">apc_store,</span> for example, and <span style="font-weight: bold;">apc_fetch</span> to get the value back without needing to execute the <span style="font-weight: bold;">expensive_function()</span> function again.<br /><br />A common technique employed by frameworks like <span style="font-style: italic;">symfony</span> is to parse some configuration files, like YAML ones, and the <span style="font-weight: bold;">var_export</span> them into a PHP file. Something like:<br /><br /><pre><br /><?php<br />$some_config = array('value_a', 'value_b');<br />?><br /></pre><br /><br />So later the framework just includes that file avoiding to parse the YAML file twice. This is technique is just file caching, but if you happen to have APC enabled, then you can benefit from the fact that the opcodes were cached by APC. So:<br /><br /><span style="font-weight: bold;">Misconception #1:</span><br /><br />That APC caches the opcodes doesn't mean that loading whatever that file has won't be expensive performance wise. As an example, let's say that the framework caches a big array of configuration values. That array has to be loaded in memory again. AFAIK that can't be avoided. So, that APC caches the opcodes, doesn't mean that reloading it comes for free.<br /><br />And what about Memcached?<br /><br /><span style="font-weight: bold;">Misconception #2:</span><br /><br />Another misconception that I've heard is that Memcached is fast and it's speed is compared with hell's speed. That doesn't mean that is faster than accessing something from the PHP process memory, which is the case with APC. Every time you retrieve a value from Memcached, it has to perform a TCP roundtrip to get that value, besides opening a TCP connection to the server per request if you are not using persistent connections.<br /><br />So while Memcached is fast, I won't recommend to use it for small values that are frequently accessed. APC can work just fine there –taking into account the size of the values of course–. So in the case of symfony, we use Memcached to cache view templates, results from queries, and things like that. But in the case of the routing generation calls, like the ones for url_for or link_to, I would prefer APC, which is what we use for the routing configuration.<br /><br />Keep in mind that this is based on my experience, so take this for what it is and of course I'll love to read your comments about this topic here.Alvarohttp://www.blogger.com/profile/05177930414107959806noreply@blogger.com5tag:blogger.com,1999:blog-6669934595599937894.post-80064375605585257152010-03-08T08:13:00.000-08:002010-03-08T08:20:19.459-08:00Erlang as a Fast Key Value Store for PHPIn this post I want to show you some of the neat things that can be done with the <a href="http://code.google.com/p/mypeb/">PHP-Erlang Bridge extension</a>: A Key Value Store.<br /><br />Erlang comes packed with a Key Value store in the form of the <a href="http://www.erlang.org/doc/man/ets.html">ETS</a> module. This is database is pretty fast and efficient for storing the Erlang terms in memory.<br /><br />I tried a proof of concept with the PHP extension and I obtained impressive results: Storing <span style="font-weight: bold;">150.000</span>+ items in the ETS in <span style="font-weight: bold;">1 second</span>! All that running on my Macbook Pro.<br /><br />What I did was to write a PHP class wrapping the calls to the Erlang ETS module like for example:<br /><pre><br />public function insert($key, $value)<br />{<br /> $x = peb_encode("[~a, {~a, ~s}]", array(array(<br /> $this->name,<br /> array($key, $value)<br /> )));<br /> $result = peb_rpc("ets", "insert", $x, $this->link);<br /> return peb_decode($result);<br />}<br /></pre>Maps to this call in Erlang:<br /><pre>> ets:insert(tablename, {key, value}).<br /></pre>You can see the full code example <a href="http://gist.github.com/324043">here</a>.<br /><br />So here are the steps:<br /><br />- Install the PEB extension from <a href="http://code.google.com/p/mypeb/source/checkout">source</a><br />- Start Erlang with this Command: <span style="font-weight: bold;">erl -sname node -setcookie abc</span><br />- Create the ETS table: <span style="font-weight: bold;">ets:new(test, [set, named_table, public]).</span><br />- Save the gist to a file and run it: <span style="font-weight: bold;">php ets.php</span><br /><br />So while this is a very simple proof of concept, I just wanted to illustrate some of the cool things that can be done with this extension. For example the speed of encoding/decoding from Erlang to PHP is pretty decent as well as the communication speed.<br /><br />Please let me know your thoughts about it in the comments.Alvarohttp://www.blogger.com/profile/05177930414107959806noreply@blogger.com0tag:blogger.com,1999:blog-6669934595599937894.post-74885251326556560942010-02-09T06:54:00.000-08:002010-02-09T07:15:25.726-08:00Meeting With Francesco Cesarini and the ECUGLast week I was invited by the <a href="http://ecug.org/">Erlang China User Group</a> to meet <a href="http://twitter.com/francescoC">Francesco Cesarini</a> from <a href="http://www.erlang-solutions.com/">Erlang Solutions</a> who was in Shanghai. Since he's the author of the <a href="http://oreilly.com/catalog/9780596518189">Erlang Programming</a> book by O'Reilly this was an amazing opportunity to learn more about Erlang from someone with real world experience in the field.<br /><br />The guys from the <span style="font-weight: bold;">ECUG</span> picked up a nice Chinese restaurant where we shared our experience about Erlang from our several points of view.<br /><br />I had my share of questions about topics such as Riak, Mnesia, RabbitMQ, Ejabberd and what not. It was nice to learn how big the Erlang world is in the enterprise, how is it used for serious matters such as banking, item traceability, and of course all the other features of Erlang, like reliability, performance, etc.<br /><br />One important topic in which we all agreed was how the language gap between English and Chinese produces two phenomena that can slow down Erlang in becoming popular in China and at the same time keeps the rest of the world unaware of what Chinese companies are doing with Erlang.<br /><br />Luckly this will start to change since the Erlang Programming book is about to be released in Chinese and the guys from the ECUG are translating some of Erlang documentation to Chinese.<br /><br />Besides that I plan to give my 2 cents by writting some blog posts in english about the Erlang movement here in China. Also we have talked with the guys from the ECUG to have some Conferences about RabbitMQ and other Erlang products that we use at the <a href="http://www.thenetcircle.com">company</a> I work for.<br /><br />To end my post I'd like to share a couple of pictures from our meeting:<br /><br /><div style="text-align: center; font-weight: bold;">Xihe Yu with Francesco Cesarini<br /></div><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKDa0aXAIffCqGrh0EBvfKzTW_K9_V2fs0Mzw04F9lYvlXq7c3gLARkfVsuNJnsx8Y8LEXk7Nkpeog3TqpgABX0Jc0EUYRGX4opHHsczbxdp0btrGu5DnHZsEy4FWnz6LArM_1joihLEJs/s1600-h/IMG_0103.jpg"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 240px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKDa0aXAIffCqGrh0EBvfKzTW_K9_V2fs0Mzw04F9lYvlXq7c3gLARkfVsuNJnsx8Y8LEXk7Nkpeog3TqpgABX0Jc0EUYRGX4opHHsczbxdp0btrGu5DnHZsEy4FWnz6LArM_1joihLEJs/s320/IMG_0103.jpg" alt="" id="BLOGGER_PHOTO_ID_5436258024406353586" border="0" /></a><br /><br /><br /><div style="text-align: center;"><span style="font-weight: bold;">The gang with Francesco Cesarini</span><br /></div><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2cfxGVie0TEK5qfhO6CuzSIZ0bvqlddPmKM3_ZetbVdQVTqEpe3sycIYSO5ZXcSgt87-SNYTcFC2RAMYQRfv9wxDfXoizN9OvxCjUu0YE1SLu91SR32zIenfZmWcED_gOiF4EjOud_9BP/s1600-h/IMG_0105.jpg"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 320px; height: 240px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2cfxGVie0TEK5qfhO6CuzSIZ0bvqlddPmKM3_ZetbVdQVTqEpe3sycIYSO5ZXcSgt87-SNYTcFC2RAMYQRfv9wxDfXoizN9OvxCjUu0YE1SLu91SR32zIenfZmWcED_gOiF4EjOud_9BP/s320/IMG_0105.jpg" alt="" id="BLOGGER_PHOTO_ID_5436258525965821522" border="0" /></a>Alvarohttp://www.blogger.com/profile/05177930414107959806noreply@blogger.com4tag:blogger.com,1999:blog-6669934595599937894.post-59546290657052965222010-02-01T05:57:00.000-08:002010-02-01T06:04:51.583-08:00Sharing Sessions Between PHP and EjabberdIn my last <a href="http://obvioushints.blogspot.com/2010/01/inspecting-php-sessions-from-python.html">post</a> I wrote about a pet project I started to share sessions between PHP an Python. In this post I want to show you how we can share the sessions between PHP and Ejabberd.<br /><br />So here's the problem. In one of the projects where we want to use XMPP we have a users databasase of around 2.5 millions users. We want that those users are able to login to our Ejabberd server using the same database. This means that every time a user logs into our site, we will query the database with PHP to see if he's allowed to login, and then ejabberd will query the<br />database again for the same purpose. Now, since the user is already authenticated in our PHP app, why don't we just share the session information with Ejabberd? Here's where InspectorD comes into play.<br /><br />The first piece that we will use to solve this problem are Ejabberd external authentication scripts. In our case instead of authenticating against a database, we will user InspectorD to check whether a user is authenticated in our website. To do this we need to find some means of passing PHP's session_id to our auth script. How to do this?<br /><br />In PHP there's a function called <a href="http://www.php.net/session_id">session_id()</a> that returns the current session_id key. We will use this string as a user password for Ejabberd, so for example, using <a href="http://code.stanziq.com/strophe/">Strophe</a> we can do something like this:<br /><br /><pre><br />connection = new Strophe.Connection(BOSH_SERVICE);<br />connection.connect(+'@someserver', , onConnect);<br /></pre><br /><br />Then Ejabberd will call our external authenticatinon script passing that <span style="font-weight: bold;">nickname</span> and the <span style="font-weight: bold;">session_id</span> as <span style="font-weight: bold;">password</span>. In our case we store the session information in Memcache, so our script will use the class <span style="font-weight: bold;">SessionInspectorMemcache</span> from InspectorD library. This class will connect to the session memcahe and from there will retrieve the session information belonging to that session_id. Finally it will return True or False depending if the user related to that session_id is authenticated or not.<br /><br />You can see the complete authentication script <a href="http://github.com/videlalvaro/InspectorD/blob/master/ejabberd_auth.py">here</a><br /><br />If you are not using memcache to store the session information then you can create a Python class that <span style="font-weight: bold;">extends</span> from InspectorD's <span style="font-weight: bold;">SessionInspector</span> class and implements the <span style="font-weight: bold;">getData</span> method. You can see an example on the SessionInspectorMemcache class.<br /><br />I hope this may result useful to you and don't hesitate to clone and improve InspectorD <a href="http://github.com/videlalvaro/InspectorD">source code</a>.<br /><br />NOTE: I did a similar script using PHP but I found it somehow harder to implement than using InspectorD code. If you want to see that code, just ask in the comments and I will post it on github.Alvarohttp://www.blogger.com/profile/05177930414107959806noreply@blogger.com1tag:blogger.com,1999:blog-6669934595599937894.post-67347860213945149762010-01-21T06:51:00.000-08:002010-01-21T06:57:21.163-08:00Inspecting PHP sessions from Python<div>For one of our PHP projects we wanted to be able to inspect the PHP sessions from outside PHP. For example we want to know the users privileges at certain moment, i.e. if the user is logged in or not. </div><div><br /></div><div>Why would you need that you may ask?</div><div><br /></div><div>Well, let's say that our symfony application stores the result of a cached action in Memcache, having two versions of the result HTML, one for logged in users and one for logged out ones. In that case we want to avoid loading symfony at all and returning directly the HTML from Nginx. One of our devs wrote a Nginx module that does just that, it gets from the Memcache certain value, if it's found, then it returns the HTML immediately, else it calls symfony to handle the request. The problem with this approach is that the Nginx doesn't know if the user is authenticated or not, so it can't handle the case where we have two different versions of HTML output for one action. Well, until now...</div><div><br /></div><div>Please welcome <a href="http://github.com/videlalvaro/InspectorD">InspectorD</a> a Python daemon that can inspect PHP sessions.</div><div><br /></div><div><b>InspectorD</b> is tcp server that understands a very simple text protocol: you ask it if certain <b>session_id</b> is authenticated and it replies <b>1</b> if it does, or <b>0</b> if it doesn't.</div><div><br /></div><div>Here's an sample session:</div><div><br /></div><div><pre></div><div><div>telnet localhost 3002</div><div>isauth oglnp9phvn8ac04obdqjk6dko3</div><div>0</div><div>isauth bj6sc485t9s46o57qpngod5lm7</div><div>1</div><div>isauth bj6sc485t9s46o57qpngod5lm7 oglnp9phvn8ac04obdqjk6dko3 n63o4uk297c49131dcdg0h7g72</div><div>1</div><div>0</div><div>1</div><div>quit</div></div><div></pre></div><div><br /></div><div>The server is based on the Twisted framework and the <a href="http://hurring.com/code/python/phpserialize/">PHPUnserialize</a> module by Scott Hurring. From the later I fixed the <b>session_decode</b> method since it wasn't working for me.</div><div><br /></div><div>For installation instructions and usage see the <a href="http://github.com/videlalvaro/InspectorD">github project page</a>.</div><div><br /></div><div>Any comments and bugs reports are welcomed.</div>Unknownnoreply@blogger.com6tag:blogger.com,1999:blog-6669934595599937894.post-44013409043679816482010-01-14T01:11:00.000-08:002010-01-16T04:13:07.364-08:00Running JLang on Snow LeopardFor this new year I started playing with <a href="http://www.jsoftware.com/">J</a>, a language that I discovered last year –but that I've never found time to play with–, while installing it, I had the problem that I didn't run on Snow Leopard.<div><br /></div><div>After asking on twitter on how to solve this problem, <a href="http://twitter.com/kaleidic">@kaleidic</a> pointed me to this <a href="http://www.jsoftware.com/jwiki/System/Installation/Mac/MacSnowLeopardInstallBug">guide</a>, which tells you to modify the Java preferences to run in 32 bit mode. What I didn't liked from that approach is that it seems to modify the general preferences of Java. After some searches I found that there's an option for the <b>java</b> executable <b>-d32</b>, that makes it run in 32 bits.</div><div><br /></div><div>To apply this option simply go to the place where you installed <b>J</b> and there edit <b>bin/jwd</b>.</div><div><br /></div><div>In that file you will find a line like this:</div><div><br /></div><div><code></code></div><code><div>java -Xss8000000 -Xdock:name=J -Xdock:icon=bin/icons/jred.icns -jar bin/j.jar "$@"</div></code><div></div><div><br /></div><div>which you have to modify to:</div><div><br /></div><div><code></code></div><code><div>java -d32 -Xss8000000 -Xdock:name=J -Xdock:icon=bin/icons/jred.icns -jar bin/j.jar "$@"</div></code><div></div><div><br /></div><div>And that's it, then simply double click on the J icon and it should work as expected.</div><div><br /></div><div><b>NOTE</b>: The line breaks that you see in the execution code are inserted by blogger, as you'll see on the <b>jwd</b> file, there are no line breaks there.</div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6669934595599937894.post-3032630554248409642009-12-15T06:59:00.000-08:002009-12-15T07:06:22.119-08:00Erlang as Session Storage for PHP<div>In the last few days I been playing with the PHP extension <a href="http://code.google.com/p/mypeb/">mypeb</a> which allows us to connect to Erlang from PHP. As a simple example to show what we can do with this extension I will create a PHP class that will be used as the session_save_hanlder for PHP. By deafult PHP stores the sessions in the file system, but if you want to share the sessions over several servers, then we have to resort to using a database or Memcached. I will like to try something different by using this class to interact with an Erlang node that will act as the in memory storage for our sessions using ETS tables.</div><div><br /></div><div>To modify the session_save_hanlder we have to call the function <a href="http://cn.php.net/manual/en/function.session-set-save-handler.php">session_set_save_handler</a> and provide there six callbacks that will be used for the following actions: opening and closing a session, reading and writing to the session, destroying the sessions and garbage collect the sessions. You can read more about this function in the PHP manual.</div><div><br /></div><div>Let's start by creating the <b>open</b> callback. In our example, we will have a method <b>ErlangSessionHandler::open</b> that will connect to the Erlang node.</div><div><br /></div><div><code></div><div>public function open($save_path, $session_name)</div><div>{</div><div> if(null === $this->link)</div><div> {</div><div> $this->link = peb_connect($this->host, $this->erlang_cookie, $this->conn_timeout);</div><div> if(!$this->link)</div><div> {</div><div> throw new Exception(sprintf("Can't connect to the erlang node %s using erlang_cookie %s", $this->host, $this->erlang_cookie));</div><div> }</div><div> }</div><div> return $this->link;</div><div>}</div><div></code></div><div><br /></div><div>There we use the function <b>peb_connect</b> that expects three parameters, the <b>host</b> to connect to, the Erlang <b>secret cookie</b> and an optional <b>connection timeout</b>. This function will return a resource identifier of the connection or false on failure. For our basic example we will define those three parameters as members of the class <a href="http://bitbucket.org/videlalvaro/erlang_php_session_storage/src/tip/ErlangSessionHandler.class.php">ErlangSessionHandler</a> like this:</div><div><br /></div><div><code></div><div>protected $host = 'server@127.0.0.1';</div><div>protected $erlang_cookie = 'ABCDEFGHI';</div><div>protected $conn_timeout = 5;</div><div></code></div><div><br /></div><div>The method <b>ErlangSessionHandler::close</b> is very straightforward:</div><div><br /></div><div><code></div><div>public function close()</div><div>{</div><div> if(is_resource($this->link))</div><div> {</div><div> peb_close($this->link);</div><div> }</div><div>}</div><div></code></div><div><br /></div><div>When called it will close the connection to the Erlang node by calling: peb_close passing the resource identifier as parameter.</div><div><br /></div><div>Then we have <b>ErlangSessionHandler::read</b></div><div><br /></div><div><code></div><div>public function read($session_id)</div><div>{</div><div> $x = peb_encode("[~s]", array(array($session_id)));</div><div> $result = peb_rpc("session_handler", "read", $x, $this->link);</div><div> $rs = peb_decode($result);</div><div> $data = $rs[0];</div><div> return is_array($data) ? '' : $data;</div><div>}</div><div></code></div><div><br /></div><div>This method will be passed the <b>$session_id</b> which we will forward to the Erlang node. To accomplish that first we need to create an Erlang Message by calling the function <b>peb_encode</b>, which expects a format string and the value we want to encode into that format. In our case we need a list which will contain our session id as only element. Once we encoded the variable we will send it to Erlang by calling <b>peb_rpc</b>. This function works similar to the Erlang rpc:call function. We need to specify the Module and the Function to call as the first two parameters. The third parameter is the message we want to send, and the last parameter is the result identifier. This function will return the result of the RPC call or false on error. The session information will be the first element of the $rs variable. In case of an error in the Erlang side, $data will be an array instead of a string, that's why we return and empty string in that case. Take into account that the session read callback must return an empty string in the case that there is no session information for the provided id.</div><div><br /></div><div>Now lets see the code for the <b>ErlangSessionHandler::write</b> method.</div><div><br /></div><div><code></div><div>public function write($session_id, $session_data)</div><div>{</div><div> $x = peb_encode("[~s, ~s]", array(array($session_id, $session_data))); </div><div> $result = peb_rpc("session_handler", "write", $x, $this->link);</div><div> unset($result);</div><div> return true;</div><div>}</div><div></code></div><div><br /></div><div>This method expects two parameters, the session id and the information to store. The code here is pretty similar to the one for <b>ErlangSessionHandler::read</b>. We encode the PHP variables as Erlang terms and we send them to the session server via <b>peb_rpc</b>.</div><div><br /></div><div>Session destroy is also similar to the implementation of read, but we call peb_rpc("session_handler", "destroy", $x, $this->link); instead of "read":</div><div><br /></div><div><code></div><div>public function destroy($session_id)</div><div>{</div><div> $x = peb_encode("[~s]", array(array($session_id)));</div><div> $result = peb_rpc("session_handler", "destroy", $x, $this->link);</div><div> unset($result);</div><div> return true;</div><div>}</div><div></code></div><div><br /></div><div>The code for ErlangSessionHandler::gc is also simple:</div><div><br /></div><div><code></div><div>public function gc($max_expire_time)</div><div>{</div><div> $x = peb_encode('[~i]', array(array($max_expirte_time)));</div><div> $result = peb_rpc("session_handler", "gc", $x, $this->link);</div><div> $rs = peb_decode($result);</div><div> return $rs;</div><div>}</div><div></code></div><div><br /></div><div>Then to use the our class as session_handler we add this to our PHP code. </div><div><br /></div><div><code></div><div>$sh = new ErlangSessionHandler();</div><div><br /></div><div>session_set_save_handler( </div><div> array($sh,"open"), </div><div> array($sh,"close"), </div><div> array($sh,"read"), </div><div> array($sh,"write"), </div><div> array($sh,"destroy"), </div><div> array($sh,"gc")</div><div>);</div><div><br /></div><div>session_start();</div><div></code></div><div><br /></div><div>Then the final piece of the puzzle is to start the Erlang Session Server that is implemented in the file <a href="http://bitbucket.org/videlalvaro/erlang_php_session_storage/src/tip/session_handler.erl">session_handler.erl</a>.</div><div><br /></div><div><code></div><div>$ erl -sname server</div><div>(server@localhost)1> c(session_handler).</div><div>(server@localhost)1> session_handler:start().</div><div></code></div><div><br /></div><div>And that's it. We can start playing with our Erlang Session Storage Server.</div><div><br /></div><div><b>NOTE</b>:</div><div><br /></div><div>First I want to make clear that this code is not meant to be used in production systems. Is just an example of what can be done with the mypeb extension. </div><div><br /></div><div>I'm planning in writing a more robust session server in Erlang using the Mnesia database as a way to provide more reliable storage. With Mnesia we can easily distribute the session data across multiple servers, and in some of them store the sessions to disc.</div><div><br /></div><div>Regarding the session save handler code, I would like to port it into the mypeb extension as native C code along with some php.ini settings that can provide the Erlang node to connect to, the secret cookie, connection timeout, etc.</div><div><br /></div><div>As a final step I would like to do a small clean up to the API of the mypeb extension.</div>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6669934595599937894.post-17580191988456300332009-10-29T06:01:00.000-07:002009-10-29T06:35:38.917-07:00Writing an Erlang PubSub Client with Exmpp<div>I've been working on a system to monitor and debug web applications remotely. One of the goals of it is to receive notifications when something wrong has happened, i.e.: the load went wild, the database is overloaded, etc. While I won't be describing the whole system in this post, I'd like to present a component of it, which is a PubSub client that connects to a XMPP server to be notified of events.</div><div><br /></div><div>In my case I've already set up an <a href="http://www.ejabberd.im/">Ejabberd</a> server, where I've configured a couple of <b>PubSub</b> nodes for testing purposes. In this tutorial we will see how to create an Erlang client to receive such notifications. </div><div><br /></div><div>Regarding the PubSub service, you can read about it <a href="http://xmpp.org/extensions/xep-0060.html">here</a>. Basically you create a <b>PubSub</b> node, and then send notifications to it. Then those users who are interested in those notification can subscribe to the node and receive them. It works similar to Twitter, where you follow someone and then you receive his messages in your timeline. </div><div><br /></div><div>Our client will be written in Erlang an will use the <a href="https://support.process-one.net/doc/display/EXMPP/exmpp+home">Exmpp</a> library. Follow the instructions on their site to see how to install it.</div><div><br /></div><div><b><span class="Apple-style-span" style="font-size: small;">NOTE</span></b><span class="Apple-style-span" style="font-size: small;">: when I first started with Exmpp the code was constantly failing. After some research I found that the problem was caused by the compilation on Snow Leopard. In case you fall into the same problem, run the configure command like this</span>: </div><div><br /></div><div><code></div><div>CC='gcc -m32' CFLAGS=-m32 LDFLAGS=-m32 ./configure </div><div></code ></div><div><br /></div><div><span class="Apple-style-span" style="font-size: small;">I got that trick from </span><a href="http://www.process-one.net/en/blogs/article/compiling_ejabberd_under_snow_leopard_with_erlang_r13b/"><span class="Apple-style-span" style="font-size: small;">here</span></a><span class="Apple-style-span" style="font-size: small;"> which also works with Exmpp.</span></div><div><br /></div><div>First we create the folder <b>pubsub_client</b> and inside we set up the structure for our project. We will follow the recommendations found <a href="http://www3.erlang.org/doc/design_principles/applications.html#7.4">here</a>: </div><div><br /></div><div>It will look like this:</div><div><br /></div><div><code></div><div>pubsub_client/</div><div> - ebin/</div><div> - include/</div><div> - priv/</div><div> - src/</div><div></code></div><div><br /></div><div>At the root of the project we will add a Makefile –the code for it is provided at the end of the post– and this shell script that will launch the Erlang Console:</div><div><br /></div><div><pre></div><div>#!/bin/sh</div><div>cd `dirname $0`</div><div>exec erl -pa $PWD/ebin -boot start_sasl -s exmpp</div><div></pre></div><div><br /></div><div>There we tell the Erlang environment to add the compiled files found inside ebin to the code load path and then it will start the exmpp application, which is required in order to use the functions from that library.</div><div><br /></div><div>Our client will be built around the <b><a href="http://erlang.org/doc/man/gen_server.html">gen_server</a></b> behavior. When launched it will connect to the Ejabberd server and wait for notifications. When they arrive we will print them to the <b>tty</b>. Also we will add a function that will let us subscribe to PubSub nodes.</div><div><br /></div><div>To start lets create a file called <b>pubsub_client.erl</b> inside the <b>src</b> folder. Then we add the following content to it:</div><div><br /></div><div><code></div><div>-module(pubsub_client).</div><div><br /></div><div>-behaviour(gen_server).</div><div><br /></div><div>-export([start/4, start_link/4, stop/0]).</div><div><br /></div><div>%% gen_server callbacks</div><div>-export([init/1, handle_call/3, handle_cast/2, handle_info/2,</div><div> terminate/2, code_change/3]).</div><div><br /></div><div>-include_lib("exmpp/include/exmpp.hrl").</div><div>-include_lib("exmpp/include/exmpp_client.hrl").</div><div><br /></div><div>-record(state, {session, jid}).</div><div></code ></div><div><br /></div><div>There we defined our module name: pubsub_client and we declared it to be a <b>gen_server</b> behavior. Then we export some functions to start and stop the client. Below are the exports of the required gen_server callbacks and we include the <b>exmpp.hrl</b> and <b>exmpp_client.hrl</b> headers, because we will need some macros that are defined there.</div><div><br /></div><div>The last line defines a <b>record</b> called <b>state</b>, which will hold our XMPP session and our JID. </div><div><br /></div><div>Lets write now the init/1 function where we will initialize our connection to the server.</div><div><br /></div><div><code></div><div>init({Host, Port, User, Password}) -></div><div> {ok, {MySession, MyJID}} = pubsub_utils:connect(Host, Port, User, Password),</div><div> {ok, #state{session=MySession, jid=MyJID}}.</div><div></code></div><div><br /></div><div>This function expects a four element tuple with the parameters to use for the connection, that is, server <b>Host</b> and <b>Port</b>, the <b>User</b> name and the <b>Password</b>. Those parameters are passed to the <b>pubsub_utils:connect/4</b> function, which will return the term {ok, Session, JID} on success. If everything its OK, our init function will return {ok, State} where State is the record that we defined before, holding our Session and our JID.</div><div><br /></div><div>Then we have to add the functions that will take care of starting and stopping the system:</div><div><br /></div><div><code></div><div>start(Host, Port, User, Password) -></div><div> gen_server:start({local, ?MODULE}, ?MODULE, {Host, Port, User, Password}, []).</div><div><br /></div><div>start_link(Host, Port, User, Password) -> </div><div> gen_server:start_link({local, ?MODULE}, ?MODULE, {Host, Port, User, Password}, []).</div><div> </div><div>stop() -></div><div> gen_server:cast(?MODULE, stop).</div><div></code></div><div><br /></div><div>The functions <b>start/4</b> and <b>start_link/4</b> both expects four parameters, which will we passed to the init function. The only difference is that <b>start_link/4</b> will link the client to our current process. You can read more about them and the whole <b>gen_server</b> behavior <a href="http://erlang.org/doc/man/gen_server.html">here</a>.</div><div><br /></div><div>Then with <b>stop/1</b> we send and asynchronous call to our module to tell it to stop. Asynchronous calls are sent using <b>gen_server:cast/2</b> and they will be processed by the <b>handle_cast/2</b> module callbacks. This are our handle_cast/2 implementations: </div><div><br /></div><div><code></div><div>handle_cast(stop, State) -> {stop, normal, State};</div><div>handle_cast(_Msg, State) -> {noreply, State}.</div><div></code></div><div><br /></div><div>In the first one we expect only the <b>stop</b> message while the second one is some sort of catch all handler. Because the return of the first one is {stop, normal, State} our client will receive a <b>terminate</b> message, which we handle in the following function:</div><div><br /></div><div><pre></div><div>terminate(_Reason, #state{session=MySession}) -> </div><div> pubsub_utils:disconnect(MySession),</div><div> ok.</div><div></pre></div><div><br /></div><div>As you can see there, first we call <b>pubsub_utils:disconnect/1</b> passing our session identifier as parameter to close our XMPP connection and the we return <b>ok</b>.</div><div><br /></div><div><b><span class="Apple-style-span" style="font-size: small;">NOTE</span></b><span class="Apple-style-span" style="font-size: small;">: In the link provided at the end, we can find the source of the </span><b><span class="Apple-style-span" style="font-size: small;">pubsub_utils</span></b><span class="Apple-style-span" style="font-size: small;"> module along with the whole project. You can read more about the implementation of the connection method </span><a href="https://support.process-one.net/doc/display/EXMPP/Scalable+XMPP+bots+with+erlang+and+exmpp"><span class="Apple-style-span" style="font-size: small;">here</span></a><span class="Apple-style-span" style="font-size: small;">.</span></div><div><br /></div><div>So far we can connect and disconnect from the Ejabberd server. Now lets write the function that will handle the notifications. </div><div><br /></div><div>When a notification is received, the Exmpp library will send a message with it to our gen_server process. When a gen_server process receives a message that was not generateed by a gen_server:cast or gen_server:call –an their similar functions– it will processed by the <b>handle_info/2</b> handler. This handler expects two parameters, <b>Info</b> and <b>State</b>, where Info is the message received which in our case willb the XMPP packet.</div><div><br /></div><div>Here's our implementation:</div><div><br /></div><div><code></div><div>handle_info(#received_packet{packet_type='message'}=Packet, State) -></div><div> process_received_packet(Packet, Fun),</div><div> {noreply, State};</div><div>handle_info(_Info, State) -> </div><div> {noreply, State}.</div><div></code></div><div><br /></div><div><b><span class="Apple-style-span" style="font-size: small;">NOTE</span></b><span class="Apple-style-span" style="font-size: small;">: We will talk later about the unbound variable Fun that you see there</span></div><div><br /></div><div>The first one expects a <b>received_packet</b> record as message. That record is defined inside the exmpp_client.hrl and is used by the Exmpp library to deliver messages. The second <b>handle_info/2</b> handler will work as a catch all handler.</div><div><br /></div><div>In our case the packet must be of the <b>message</b> <b>type</b>, which we specify in our pattern matching declaration. If so we delegate the processing to the <b>process_received_packet/2</b> function.</div><div><br /></div><div>Now is time to process the notification, which in our case means printing them to the tty. Here's the code:</div><div><br /></div><div><code></div><div>process_received_packet(#received_packet{raw_packet=Raw}, Fun) -></div><div> Event = exmpp_xml:get_element(Raw, ?NS_PUBSUB_EVENT, 'event'),</div><div> Items = exmpp_xml:get_element(Event, 'items'),</div><div> exmpp_xml:foreach(Fun, Items),</div><div> ok.</div><div></code></div><div><br /></div><div>To accomplish this we have to parse the incoming packet. Our <b>process_received_packet/2</b> function will take care of that. In its declaration we extract the Raw packet via pattern matching. Then first we use the <b>exmpp_xml:get_element/3</b> function to get the received PubSub event. This helper function from the Exmpp library expects the XML element from where we will extract the child node, the namespace the child should have -?<b>NS_PUBSUB_EVENT</b> in our case– and the name of the XML element of the child node. In our example we want an <b>event</b> node. </div><div><br /></div><div>Then from that XML element we extract the <b>Items</b>, which is what we are interested in. The items node will have one or several children containing the notification we expect inside an item node –note the plural difference–.</div><div><br /></div><div>Once we have the <b>Items</b> we pass them to the <b>exmpp_xml:foreach/2</b> function along with the <b>Fun</b> parameter. The <b>exmpp_xml:foreach/2</b> will iterate over the child elements of Items and will apply to them the anonymous function contained in Fun. In case you need it, the first argument passed to the anonymous function will be the original XML element. For this to work we have to pass this Fun to to <b>process_received_packet/2</b>. </div><div><br /></div><div>Let's declare that <b>fun</b> inside the <b>init/1</b> function and add them to the <b>state</b> of the process. This will be our new init/1 function:</div><div><br /></div><div><code></div><div>init({Host, Port, User, Password}) -></div><div> {ok, {MySession, MyJID}} = pubsub_utils:connect(Host, Port, User, Password),</div><div> Fun = fun(_XML_Element, Child) -> </div><div> case exmpp_xml:get_element(Child, 'log') of</div><div> undefined -> not_a_notification;</div><div> Log -> </div><div> io:format("Notification: ~s~n", [exmpp_xml:get_cdata_as_list(Log)])</div><div> end</div><div> end,</div><div> {ok, #state{session=MySession, jid=MyJID, on_message=Fun}}.</div><div></code></div><div><br /></div><div>Our fun will take two parameters as we discussed and it will print to the tty the log using <b>io:format/2</b>. There's some XML processing in place there, to extract our log text out. This fun will be added to our process State inside the new <b>on_message</b> field of our state record, which we will have to modify for this to work:</div><div><br /></div><div><code></div><div>-record(state, {session, jid, on_message}).</div><div></code></div><div><br /></div><div>Once we have that code in place we change the header of our <b>handle_info/2</b> function so we can extract the Fun out of the state record:</div><div><br /></div><div><code></div><div>handle_info(#received_packet{packet_type='message'}=Packet, #state{on_message=Fun}=State) -></div><div></code></div><div><br /></div><div>The final piece of our puzzle is to actually subscribe to a node, here's what we can do. We will add a function to our process API that will do this for us:</div><div><br /></div><div><code></div><div>subscribe(Service, Node) -></div><div> gen_server:call(?MODULE, {subscribe, Service, Node}).</div><div></code></div><div><br /></div><div>And we export it adding the following line after the gen_server export callbacks:</div><div><br /></div><div><code></div><div>-export([subscribe/2]).</div><div></code></div><div><br /></div><div>Our function <b>subscribe/2</b> will send a synchronous call to our process passing two parameters. The PubSub service <b>Name</b> and the <b>Node</b>. The next step is to implement the <b>handle_call/3</b> callback:</div><div><br /></div><div><code></div><div>handle_call({subscribe, Service, Node}, _From, #state{session=MySession, jid=MyJID}=State) -></div><div> IQ = exmpp_client_pubsub:subscribe(exmpp_jid:to_list(MyJID), Service, Node),</div><div> PacketId = exmpp_session:send_packet(MySession, exmpp_stanza:set_sender(IQ, MyJID)),</div><div> PacketId2 = erlang:binary_to_list(PacketId),</div><div> Reply = </div><div> receive</div><div> #received_packet{id=PacketId2, raw_packet=Raw} -></div><div> case exmpp_iq:is_error(Raw) of</div><div> true -> error;</div><div> _ -> ok</div><div> end</div><div> end,</div><div> {reply, Reply, State};</div><div></code></div><div><br /></div><div>There we use the helper function from the Exmpp library to generate an <b>IQ</b> stanza that will be sent to the XMPP server to subscribe our user. The interesting part of that code is the receive block. Let's review it. When we send an IQ packet to the server it will reply to us with another IQ packet. In order to track the IQ requests and responses, the XMPP protocol adds an <b>id</b><b> attribute</b> to the stanza. That's why we only wait for messages sent to our process that contain that packet <b>id</b>. You may be wondering why we have the receive block here. We want to know if our subscription succeeded or if it failed. If we don't have any receive block there, then the reply will be handled by the <b>handle_info/2</b> function. There will be quite hard to track the IQs ids and match them. In our simple case could be easy, but in a more complex scenario we may encounter problems when our process is receiving several messages concurrently.</div><div><br /></div><div>To try our module we can do the following sequence at the command line:</div><div><br /></div><div><code></div><div>cd /path/to/our/project</div><div>make</div><div>./start-dev.sh</div><div></code></div><div><br /></div><div>There we compile the code and launch the Erlang console. Then inside the console we input the following:</div><div><br /></div><div><code></div><div>pubsub_publisher:start("localhost", 5222, "publisher", "password").</div><div>pubsub_publisher:create_node("pubsub.localhost", "logs").</div><div></code></div><div><br /></div><div>Then we can open a new terminal window an do the following</div><div><pre></div><div>cd /path/to/our/project</div><div>./start-dev.sh</div><div>pubsub_client:start("localhost", 5222, "tutorial", "password").</div><div>pubsub_client:subscribe("pubsub.localhost", "logs").</div><div></pre></div><div><br /></div><div>Then we can go back to the first window and issue this command:</div><div><br /></div><div><pre></div><div>pubsub_publisher:send_message("pubsub.localhost", "some notification", "logs").</div><div></pre></div><div><br /></div><div>If everything worked well, then we should see the message being displayed where the <b>pubsub_client</b> is running.</div><div><br /></div><div><b>BONUS</b>: Display the log message as a <b>Growl</b> notification</div><div><br /></div><div>You have the right to wonder why in the code above we pass the anonymous function around in the process state. I did that, because later we can hook a new function to process the notifications, say, to send them directly to Growl :) –for the non Mac users, Growl is a free application for Mac that can display notifications on our screen more info <a href="http://growl.info/">here</a>–.</div><div><br /></div><div>If you review the code provided at the end of this post, there's a function <b>pubsub_client:use_growl/0</b> that when called will swap the log handler and instead of displaying the notifications in the <b>tty</b>, it will forward them to <b>Growl</b>. Since the implementation is easy to understand I won't explain it here.</div><div><br /></div><div>Thanks for reading this far and I hope this tutorial will push you to create some nice erlang applications using Exmpp and PubSub.</div><div><br /></div><div>PubSub client <a href="http://bitbucket.org/videlalvaro/pubsub_client/wiki/Home">Code</a>.</div>Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6669934595599937894.post-7949592424742842742009-09-01T05:25:00.000-07:002010-03-22T09:24:56.559-07:00Running Haskell GHC on Snow Leopard***UPDATE 2009/03/23***<br /><br />There's a new version of the Haskell platform that works with Snow Leopard. If you install that then you shouldn't need to use the tip explained here. Please comment if it works for you<br /><br />http://hackage.haskell.org/platform/mac.html<br /><br />***UPDATE 2009/03/23***<br /><br />After so much time spent trying to get a working GHC in my Mac with Snow Leopard I could arrive to the solution. Here it is, pretty simple:<div><br /></div><div>1 - Download Haskell Platform for Mac: <a href="http://hackage.haskell.org/platform/">http://hackage.haskell.org/platform/</a></div><div>2 - Install GHC and the Haskell Platform that are inside the .dmg.</div><div>3 - Patch your ghc compiler to produce 32 bits code adding the following options:</div><pre>-optc-m32 -opta-m32 -optl-m32<br /></pre><pre><span class="Apple-style-span" style="white-space: normal;font-family:Georgia,serif;font-size:16px;" >The ghc script that you have to patch is located here: <b>/usr/bin/ghc</b></span></pre><pre><span class="Apple-style-span" style="white-space: normal;font-family:Georgia,serif;font-size:16px;" >Open it with your favorite text editor and add those options. This at least will let you install new packages and compile your code with no problems until the Haskell team release a new version of GHC for mac.</span></pre><pre><span class="Apple-style-span" style=";font-family:Georgia,serif;font-size:130%;" ><span class="Apple-style-span" style="white-space: normal;font-size:16px;" >Most of the tricks for this I got them from here: <a href="http://www.nabble.com/Snow-Leopard-Breaks-GHC-td25198347.html">http://www.nabble.com/Snow-Leopard-Breaks-GHC-td25198347.html</a></span></span></pre>Unknownnoreply@blogger.com15tag:blogger.com,1999:blog-6669934595599937894.post-59976705681567434792009-07-06T12:24:00.000-07:002009-07-06T12:46:51.813-07:00My Guess on Symfony 2<div><div>After I read <a href="http://twitter.com/fabpot/statuses/2454825809">this</a> tweet from <a href="http://fabien.potencier.org/">Fabien</a> I was left thinking on how Symfony 2 will be. Then I</div><div>remembered <a href="http://fabien.potencier.org/talk/22/phpquebec-2F009-symfony-2">this</a> presentation from Fabien where he talks about the new framework. I took a look at it and then I decide to glue the pieces together to get something working out of the</div><div>code given in slide 28.</div><div><br /></div><div>So to start with it, I needed an application to build. Since some days ago I'm playing with <a href="http://www.mongodb.org/">MongoDB</a>, a document oriented database. To learn how to use it I built a centralized logger for symfony applications. The idea is that if we have in production twenty machines serving symfony and then we need to parse the logs to find possible errors, etc., it will be nice to have a tool that centralizes the logs in one place. Since this database is lightweight and fast, I wrote a simple logger to store the messages in a MongoDB database instead of using the normal file logger.</div><div><br /></div><div>After I got the <a href="http://code.google.com/p/mongodbloganalyzer/source/browse/#svn/trunk/sfMongoDBLoggerPlugin%3Fstate%3Dclosed">logger</a> working I needed a way to display and search through the logs. Initially I built a symfony application that was able to filter the logs by priority and by some words in the log message. The feeling I got was that a full symfony 1.2 project was too much for such a</div><div>simple web app. This was the perfect excuse to experiment with Symfony 2.</div><div><br /></div><div><b>The Logger</b></div><div><br /></div><div>The MongoDB logger is just a simple symfony logger that stores for every symfony log an array with this structure:</div><div><br /></div><div><pre> </pre></div><div><br /></div><div>$log = array( 'type' => $type, 'message' => $message,</div><div> 'time' => time(), 'priority' => $priority) ); </div><div></div><div><br /></div><div>The idea is to provide a form to issue queries to the database to filter the logs by any of those fields. i. e.: I type <b>sfRouting </b>and I should see only those logs that contain that word in their messages.</div><div><br /></div><div>Here's a screen shot of the final application:</div></div><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFL1Kk0MHOiR-nW6OcKwHAD8MwjhltH9oh-pNvUZD-R9Sa4ooxbp4ZI5kuqmdzngWp_OJeZcvMcDROG-tF8FcRnSClGKka-6j1O-7Rm4uK9cxBd8aNCe24wIUAENyw7F_Wc7rd1qGCgVA/s1600-h/LogAnalyzer.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 234px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFL1Kk0MHOiR-nW6OcKwHAD8MwjhltH9oh-pNvUZD-R9Sa4ooxbp4ZI5kuqmdzngWp_OJeZcvMcDROG-tF8FcRnSClGKka-6j1O-7Rm4uK9cxBd8aNCe24wIUAENyw7F_Wc7rd1qGCgVA/s400/LogAnalyzer.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5355430784290151106" /><br /></a><div><div>So, how to build that using Symfony 2?</div><div><br /></div><div>First we create the folder structure like this:</div><div><br /></div><div><pre> </pre></div><div>-/ </div><div>--/ apps </div><div>--/ config </div><div>--/ lib </div><div>--/ web </div><div></div><div><br /></div><div>Inside <b>web</b> we place the <b>index.php</b> file which has the following content:</div><div><br /></div><div><pre></pre></div><div><div><br /></div><div>define('ROOT_PATH', dirname(__FILE__).'/..');</div><div><br /></div><div>require_once ROOT_PATH . '/config/sf_requires.php';</div><div>require_once ROOT_PATH . '/config/app_requires.php';</div><div><br /></div><div>$app = new LogAnalyzer(); $app->run()->send(); </div><div></div><div><br /></div><div>There we define the root path, and then we include two configuration files, one which will take care of requiring the Symfony libraries and the other that will require the application files.</div><div><br /></div><div>Then we instantiate the application class and we run it.</div><div><br /></div><div>Now let's check what's inside the <b>sf_requires.php</b> file:</div><div><br /></div><div><pre></pre></div><div>define('SF_LIB_PATH', ROOT_PATH . '/lib/vendor/symfony/lib');</div><div><br /></div><div>require_once SF_LIB_PATH . '/utils/sfToolkit.class.php';</div><div>require_once SF_LIB_PATH . '/utils/sfParameterHolder.class.php'; </div><div>require_once SF_LIB_PATH . '/event_dispatcher/sfEventDispatcher.php';</div><div>require_once SF_LIB_PATH . '/request/sfRequestHandler.class.php'; </div><div>require_once SF_LIB_PATH . '/request/sfRequest.class.php'; </div><div>require_once SF_LIB_PATH . '/request/sfWebRequest.class.php'; </div><div>require_once SF_LIB_PATH . '/response/sfResponse.class.php'; </div><div>require_once SF_LIB_PATH . '/response/sfWebResponse.class.php';</div><div></div><div><br /></div><div>First we define the location of the Symfony libraries and then we proceed to include the required files.</div><div><br /></div><div>The only new class here is the <b>sfRequestHandler</b> which Fabien describes in his presentations, the other ones I just took from a <b>symfony 1.3</b> distrubution.</div><div><br /></div><div>With those files included, we are done with what refers to symfony, then we have to include the application files. So the contents of <b>app_requires.php</b> will be:</div><div><br /></div><div><pre></pre></div><div>require_once ROOT_PATH . '/lib/dba/MongoLogReader.class.php'; </div><div>require_once ROOT_PATH . '/lib/dba/CollectionModel.class.php'; </div><div>require_once ROOT_PATH . '/apps/LogAnalyzer.class.php';</div><div></div><div><br /></div><div>Besides the <b>LogAnalyzer</b> class we include two classes that will take care of querying the log database. As we can see, the LogAnalyzer class will reside under the apps folder and then others under lib/dba.</div><div><br /></div><div>So now let's check what's inside the LogAnalyzer class.</div><div><br /></div><div><pre> </pre></div><div>public function __construct() </div><div>{ </div><div> $this->dispatcher =</div><div> new sfEventDispatcher();</div><div> $this->dispatcher->connect('application.load_controller', array($this, 'loadController'));</div><div>}</div><div></div><div><br /></div><div>On instantiation we create a new instance of the <b>sfEventDispatcher</b> and we connect our application to the <b>application.loadController</b> event which will be fired by the <b>sfRequestHandler::handleRaw()</b> method. There we tell it that the <b>loadController</b> method of our application will process the request.</div><div><br /></div><div>Then we have the run method:</div><div><br /></div><div><pre></pre></div><div>public function run() </div><div>{ </div><div> $request = new sfWebRequest($this->dispatcher); </div><div> $handler = new sfRequestHandler($this->dispatcher); </div><div> $response = $handler->handle($request); </div><div> return $response; </div><div>}</div><div></div><div><br /></div><div>Here we initialize a <b>sfWebRequest</b> object to start parsing the request parameters. Then we instantiate our <b>sfRequestHandler</b> and we call the handle method. The handle method will return a response object, which is the one where we call <b>send()</b> in the <b>index.php</b> file to output the response to the browser.</div><div><br /></div><div>When the <b>sfRequestHandler</b> start to do it's job it will fire the <b>application.load_controller</b> event for which we set up the following listener:</div><div><br /></div><div><pre></pre></div><div>public function loadController(sfEvent $event) </div><div>{ </div><div> $event->setReturnValue(array(array($this, 'execute'), array($this->dispatcher, <span class="Apple-tab-span" style="white-space:pre"> </span>$event['request']))); </div><div> return true; </div><div>}</div><div></div><div><br /></div><div>There we say that the method execute of the LogAnalyzer class will take care of generatiing the response data out of the request.</div><div><br /></div><div>And finally the code for the execute method:</div><div><br /></div><div><pre></pre></div><div>public function execute($dispatcher, $request)</div><div>{</div><div> $response = new sfWebResponse($dispatcher); </div><div> $response->setContent($this->render($this->getTemplateValues($request))); </div><div> return $response;</div><div>}</div><div></div><div><br /></div><div>There we instantiate a <b>sfWebResponse</b>. The content of this one will be the result of the proteced method render. –Here I must say that is possible to create our own View class to</div><div>handle this part, but for this example I preffer to build it like this–.</div><div><br /></div><div><pre></pre></div><div>protected function render($values)</div><div>{</div><div> extract($values);</div><div> ob_start();</div><div> ob_implicit_flush(0);</div><div> require(ROOT_PATH . '/apps/template.php');</div><div> return ob_get_clean();</div><div>}</div><div></div><div><br /></div><div>This method expects an array with all the variables that will be used in the template. This values are extracted from the array and inserted in the current scope by calling the extract function. As we can see, the template is a <a href="http://code.google.com/p/mongodbloganalyzer/source/browse/trunk/loganalyzer/apps/template.php">simple php</a> file that is required from the apps folder. This file is plain PHP code embedded into HTML.</div><div><br /></div><div>The <b>getTemplateValues</b> method take will get the data out of the database, plus interpreting the request:</div><div><br /></div><div><pre></pre></div><div>protected function getTemplateValues($request)</div><div>{</div><div> $values = array();</div><div> </div><div> $values['sf_request'] = $request;</div><div> </div><div> $values['collections'] = $this->getCollections();</div><div> </div><div> $values['priorities'] = $this->getPriorities();</div><div> </div><div> $values['cursor'] = $this->getLogs($request);</div><div> $values['pageNumber'] = $request->getParameter('page', 1); </div><div> $values['cursor']->skip(($values['pageNumber'] - 1) * $this->maxPerPage)->limit($this->maxPerPage);</div><div> $values['hasMore'] = $values['cursor']->count() > ($this->maxPerPage * $values['pageNumber']);</div><div> $values['filterParams'] = $this->buildFilterParams($request);</div><div> </div><div> return $values;</div><div>}</div><div></div><div><br /></div><div>And that's it! Which such a simple structure we can leverage the power of the <b>sfRequestHandler</b> class which will be at the core of the new Symfony version. We know that symfony does very well for complex projects, but sometimes I felt like it was too big for a simple application like this one. With this new component I think that this distinction will be gone.</div><div><br /></div><div><b>THE CODE</b>:</div><div><br /></div><div>The <b>application code</b> and the <b>sfMongoDBLogger</b> class can be found <a href="http://code.google.com/p/mongodbloganalyzer/">here</a>.</div><div><br /></div><div>You will need to setup a virtual host in order to run this application.</div><div><br /></div><div><b>RESOURCES/REQUIREMENTS</b>:</div><div><br /></div><div>To learn more on MongoDB refer to this website: <a href="http://www.mongodb.org/">http://www.mongodb.org</a> </div><div>If you setup MongoDB as explained here http://www.mongodb.org/display/DOCS/Getting+Started, you should be able to run this project without problems –Give it a try, MongoDB is pretty easy to setup and the documentation is very good–</div><div>For the installation instructions of the PHP native driver go <a href="http://www.mongodb.org/display/DOCS/Installing+the+PHP+Driver">here</a></div><div><br /></div><div><b>IMPORTANT</b>:</div><div><br /></div><div>Even if this should be implicit, keep in mind that this are my personal views on the subjects. This is by no means an official statement from the Symfony project. Is just what I believe this new component will be based on Fabien presentation.</div><div style="text-align: center;"><span class="Apple-style-span" style="color:#0000EE;"><span class="Apple-style-span" style="text-decoration: underline; "><br /></span></span></div></div></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6669934595599937894.post-82346458972515696922009-07-02T19:48:00.000-07:002011-11-05T14:55:35.149-07:00Benchmarking MongoDB VS. Mysql<div><b>EDIT November 2011: </b>Please take this benchmarks with a grain of salt. They are completely non-scientific and when choosing a data store probably raw speed is not all you care about. Try to understand the trade-offs of each database, learn about how your data will be handled in case of losing a server. What happens if there's data corruption? How replication works in your specific database… and so on. After all… don't choose a database over another just because of this blogpost.</div><div><br /></div><div>One of the projects I work for at the company has a message system with more than 200 million entries in a MySQL database. We've started to think about how can we scale this further. So in my research for alternatives look at nosql databases. In this post I would like to share some benchmarks I ran against <a href="http://www.mongodb.org/">Mongodb</a> –a document oriented database– compared to MySQL.</div><div><br /></div><div>To perform the tests I installed MongoDB as explained here. I also installed PHP, MySQL and Apache2 from Macports. I did no special configurations in any of them.</div><div><br /></div><div>The hardware used for the tests is my good ol' Macbook White:</div><div><br /></div><div> Model Name:<span class="Apple-tab-span" style="white-space:pre"> </span>MacBook</div><div> Model Identifier:<span class="Apple-tab-span" style="white-space:pre"> </span>MacBook2,1</div><div> Processor Name:<span class="Apple-tab-span" style="white-space:pre"> </span>Intel Core 2 Duo</div><div> Processor Speed:<span class="Apple-tab-span" style="white-space:pre"> </span>2 GHz</div><div> Number Of Processors:<span class="Apple-tab-span" style="white-space:pre"> </span>1</div><div> Total Number Of Cores:<span class="Apple-tab-span" style="white-space:pre"> </span>2</div><div> L2 Cache:<span class="Apple-tab-span" style="white-space:pre"> </span>4 MB</div><div> Memory:<span class="Apple-tab-span" style="white-space:pre"> </span>4 GB</div><div> Bus Speed:<span class="Apple-tab-span" style="white-space:pre"> </span>667 MHz</div><div> Boot ROM Version:<span class="Apple-tab-span" style="white-space:pre"> </span>MB21.00A5.B07</div><div> SMC Version (system):<span class="Apple-tab-span" style="white-space:pre"> </span>1.13f3</div><div></div><div><br /></div><div>Because I don't have enough space to store the MongoDB database in my local hardrive, I launched the server with this command:</div><div><pre></pre></div><div><b>./bin/mongod --dbpath '/Volumes/alvaro/data/db/'</b></div><div></div><div><br /></div><div>which tells <b>MongoDB</b> to <b>use</b> my <b>USB hardrive</b>. YES, my USB hardrive :-P</div><div><br /></div><div>The MySQL server stored the data in the local hardrive.</div><div><br /></div><div>What was the test?</div><div><br /></div><div>I loaded in both databases 2 million records from our real data of the message system. Every record has 28 columns, holding informatin about the sender of the message and the recipient, plus the subject, date, etc. For MySQL I used <b>mysqldump. </b>For MongoDB I used the following:</div><div><br /></div><div><div><pre></pre></div><div>$query = "SELECT * from messsage"; </div><div>$result = mysql_query($query); </div><div>while($row = mysql_fetch_assoc($result))</div><div>{</div><div> $collection->insert($row);</div><div>}</div><div></div><div><br /></div><div>Of course that for the real data loading I added some paginations, I didn't retrieved 2M records at once. And there was some code to initialize the MySQL connection and Mongo to get that $collection object.</div><div>The MySQL databases had index on the sid and tid fields (sender id and target id), so I added them to the MongoDB database.</div></div><div><pre></pre></div><div>$m = new Mongo();</div><div>$collection = $m->selectDB("msg")->selectCollection("msg_old");</div><div>$collection->ensureIndex( array("sid", "tid") );</div><div></div><div><br /></div><div>Then I wrote some <a href="http://code.google.com/p/mongodbvsmysql/">simple</a> code that will select a limit of 20 records filtered by sid. In the real application this means I'm watching the first 20 messages of my outbox.</div><div><br /></div><div><b>EDIT - 2009/07/03</b></div><div><br /></div><div>Due to some confusion I have to make something clear. What I'm benchmarking is not the data loading into both databases, nor traversing the data, etc., but the code that you can find <b><a href="http://code.google.com/p/mongodbvsmysql/">here</a> </b>. </div><div><br /></div><div>This is a similar case of what a user message outbox (or inbox) could be in a production website. The users access his inbox and we retrieve up to 20 messages of his inbox, which are then displayed in an html table. What <b>siege</b> accessed was a script serving that html generated out of the query results.</div><div><br /></div><div>So the idea is, if MongoDB or MySQL are the backends of this message system, which one will be faster for <b>this specific use case</b>. This benchmark is not about if MongoDB is better than MySQL for every use case out there. We use MySQL a lot in production and we will keep using it as far as I can tell. And yes, I know that MySQL and MongoDB are two totally different technologies that probably only share the word database in their descriptions.</div><div><br /></div><div><b>END EDIT - 2009/07/03</b></div><div><br /></div><div>I did the code for Mongodb and MySQL. Then my idea was to launch <a href="http://www.joedog.org/index/siege-home">siege</a> and pick some random user ids from a text file and do the stress tests.</div><div><br /></div><div>Here's an extract from the url textfile: </div><div><span class="Apple-style-span" style=" white-space: pre; font-family:monospace, fantasy;font-size:13px;"> </span></div><div>http://mongo.al/index.php?id=96</div><div>http://mongo.al/index.php?id=105</div><div>http://mongo.al/index.php?id=108</div><div>http://mongo.al/index.php?id=113</div><div>http://mongo.al/index.php?id=116</div><div>http://mongo.al/index.php?id=117</div><div>http://mongo.al/index.php?id=127</div><div>http://mongo.al/index.php?id=129</div><div>http://mongo.al/index.php?id=130</div><div>http://mongo.al/index.php?id=134</div><div></div><div><br /></div><div>This means that siege will pick a random url and hit the server, requesting the outbox of that user id.</div><div><br /></div><div>Then I increased the ulimit to be able to run this test:</div><div><pre></pre></div><div><b>siege -f ./stress_urls.txt -c 300 -r 10 -d1 -i</b></div><div></div><div><br /></div><div>With that command I launch siege, telling it to load the urls to visit form the text file. It will simulate 300 concurrent users and will do 10 repetitions with a random delay between 0 and 1. The last option tells siege to work in internet mode, so it will pick urls randomly from the text file.</div><div><br /></div><div>When I launched the test wit MongoDB as backend it worked without problems. With the MySQL it crashed quite often. Below I add the results I obtained for both of them.</div><div><br /></div><div><b>MongoDB test results:</b></div><div><br /></div><div>siege -f ./stress_urls.txt -c 300 -r 10 -d1 -i</div><div><br /></div><div>Transactions: 2994 hits</div><div>Availability: 99.80 %</div><div>Elapsed time: 11.95 secs</div><div>Data transferred: 3.19 MB</div><div>Response time: 0.26 secs</div><div>Transaction rate: 250.54 trans/sec</div><div>Throughput: 0.27 MB/sec</div><div>Concurrency: 65.03</div><div>Successful transactions: 2994</div><div>Failed transactions: 6</div><div>Longest transaction: 1.47</div><div>Shortest transaction: 0.00</div><div></div><div><br /></div><div><b>MySQL tets results:</b></div><div><pre></pre></div><div>siege -f ./stress_urls_mysql.txt -c 300 -r 10 -d1 -i</div><div><br /></div><div>Transactions: 2832 hits</div><div>Availability: 94.40 %</div><div>Elapsed time: 23.53 secs</div><div>Data transferred: 2.59 MB</div><div>Response time: 0.74 secs</div><div>Transaction rate: 120.36 trans/sec</div><div>Throughput: 0.11 MB/sec</div><div>Concurrency: 89.43</div><div>Successful transactions: 2832</div><div>Failed transactions: 168</div><div>Longest transaction: 16.36</div><div>Shortest transaction: 0.00</div><div></div><div><br /></div><div>As we can see, <b>MongoDB</b> performed more than <b>2X better</b> than MySQL for this specific case. And remember, MongoDB was reading the data from my USB hardrive ;-).</div>Unknownnoreply@blogger.com6tag:blogger.com,1999:blog-6669934595599937894.post-5723421525015491052009-06-04T02:17:00.000-07:002009-06-04T02:34:15.048-07:00New Firesymfony Release<p>I'm pleased to announce the release of <strong>version 1.1</strong> of <strong>Firesymfony</strong>. This time it has a new design that we believe improves the user experience. The design was made by my colleague <a href="http://twitter.com/jacquelinewan">Jacqueline Wan</a> and the logo by Olaf Horstmann.</p>Bellow you can see some screenshots and here you have the urls to update the symfony plugins and the Firebug extension:<br /><p><a href="http://www.symfony-project.org/plugins/firesymfonyPlugin">Symfony Plugin</a></p><p><a href="http://addons.mozilla.org/en-US/firefox/addon/9096/">FB Extension</a></p><p align="center"><strong>About Panel:</strong><br /></p><p><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrkcWoSWYolyGJ1899XXpLGAo13ueLFvFgECsmW8dnnLw4szyKa5pRZnsAD8kXPpztICEpMUoNr1Rinkq3iGZpJZJPiWJ4k9cUQ34B_xnBPD2yQOW8MW8M99UZSbsNaTzK5jTtp1eRvGE/s1600-h/about.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 133px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrkcWoSWYolyGJ1899XXpLGAo13ueLFvFgECsmW8dnnLw4szyKa5pRZnsAD8kXPpztICEpMUoNr1Rinkq3iGZpJZJPiWJ4k9cUQ34B_xnBPD2yQOW8MW8M99UZSbsNaTzK5jTtp1eRvGE/s400/about.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5343400893886695714" /></a></p><p align="center"><strong>Configuration and Variables Panel:</strong></p><p><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi73EM_-tAsgkC1vg3wb2Qk6s7tdBlXYq_ld6OA31LNqvTxtXFqAGfSQDTcz76YiMb3xrRx238CcFGS8EaNMfAEyCNvvSAPwY96tfgmAjdLW2UW_Ww6IfTcWOdymeN7UQHE6Fi3e0Yt6sI/s1600-h/config.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 133px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi73EM_-tAsgkC1vg3wb2Qk6s7tdBlXYq_ld6OA31LNqvTxtXFqAGfSQDTcz76YiMb3xrRx238CcFGS8EaNMfAEyCNvvSAPwY96tfgmAjdLW2UW_Ww6IfTcWOdymeN7UQHE6Fi3e0Yt6sI/s400/config.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5343400900117497314" /></a><br /></p><p align="center"><strong>Logs Panel:</strong></p><p><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO1KuNIj1vUpuYLQT0pXATxTW3IQr1f4ct9rbOlGWzlQeycqtluCXXbZI6kt8YCv45AnNRL8bJeCUUhY1101K6jXDqi33QwG-hDcw9fP0M2m-kXqL9psJviceEfJT-F0Re7UufF63Zlmw/s1600-h/logs.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 133px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO1KuNIj1vUpuYLQT0pXATxTW3IQr1f4ct9rbOlGWzlQeycqtluCXXbZI6kt8YCv45AnNRL8bJeCUUhY1101K6jXDqi33QwG-hDcw9fP0M2m-kXqL9psJviceEfJT-F0Re7UufF63Zlmw/s400/logs.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5343400900721835586" /></a></p><p align="center"><strong>Cache Panel:</strong><br /></p><p><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYLJVK0XBeXhWhrmlmQLK8oDmvtH6b-lcOhj7DfES4MMtRJnRrBF-D5xfybwtJxJu3gypIyfgFdo9gBwkQewZljHzqhNt-lG_ZOZRqwCnrlPIy8MOq2_CtRtM6D3RW7YeqoLcYvMHbvUM/s1600-h/cache.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 133px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYLJVK0XBeXhWhrmlmQLK8oDmvtH6b-lcOhj7DfES4MMtRJnRrBF-D5xfybwtJxJu3gypIyfgFdo9gBwkQewZljHzqhNt-lG_ZOZRqwCnrlPIy8MOq2_CtRtM6D3RW7YeqoLcYvMHbvUM/s400/cache.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5343400902421408258" /></a></p><p align="center"><strong>Database Panel:</strong><br /></p><p><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi840Myr79uP5SQ7J6WScU4LtAC476sy33oDB0_qHIGp3P2gtxF1SW8VAKsypKeNML67tfnFryb7tCQIVgEOWhVEpE71G2ff2sWl9eVqHA93v7fYimGYbWyplBhYZQ81k1q4tR7XAWWprg/s1600-h/db.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 133px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi840Myr79uP5SQ7J6WScU4LtAC476sy33oDB0_qHIGp3P2gtxF1SW8VAKsypKeNML67tfnFryb7tCQIVgEOWhVEpE71G2ff2sWl9eVqHA93v7fYimGYbWyplBhYZQ81k1q4tR7XAWWprg/s400/db.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5343400905711009170" /></a></p><p align="center"><strong>Timers Panel:</strong><br /></p><p><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX9zZk53sDE3Tt3QbZiXpUGlTOAoXp5jLjAIr2D-kj4h_ZwtpZpltclf_kAiK58N5bAYTQrdFhaPghi8OtF0rLqMePGBtUKworysRbzqtpC429BPXzwlNky-7WoOcfpopbxZtb64ZitmQ/s1600-h/timers.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 133px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX9zZk53sDE3Tt3QbZiXpUGlTOAoXp5jLjAIr2D-kj4h_ZwtpZpltclf_kAiK58N5bAYTQrdFhaPghi8OtF0rLqMePGBtUKworysRbzqtpC429BPXzwlNky-7WoOcfpopbxZtb64ZitmQ/s400/timers.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5343401613496895858" /></a></p><p align="center"><strong>Information Panel:</strong><br /></p><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlLLZjyJoy88EHGP1ymIdkV2hPbknhSt8BYDztIEn7CPX2yZeGKD_AOUV_Cz0A5nTipKzub7m64IDdPEirqhDK16N7XS5GVqaJbbq0hKLbtUaldDY37JEj4P7QbHhNczMN1OYxqxrfpz8/s1600-h/info.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 133px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlLLZjyJoy88EHGP1ymIdkV2hPbknhSt8BYDztIEn7CPX2yZeGKD_AOUV_Cz0A5nTipKzub7m64IDdPEirqhDK16N7XS5GVqaJbbq0hKLbtUaldDY37JEj4P7QbHhNczMN1OYxqxrfpz8/s400/info.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5343401605215313202" /></a>Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6669934595599937894.post-37850242170752206282009-03-30T06:28:00.000-07:002009-03-30T07:07:52.918-07:00How I would like to use Propel and Memcache<div>In this article I will like to share some ideas that are wandering in my mind but I haven't implemented yet. So beware!<br /></div><div><br /></div><div>I was thinking of a way to reduce database load by caching some results inside Memcache -idea which has nothing special or revolutionary this days-. <a href="http://code.google.com/p/redis/wiki/TwitterAlikeExample">This article</a> gave me some ideas that I would like to see implemented in some of the projects I work for.</div><div><br /></div><div>The code is for using mostly inside a Symfony/Propel project, but could be adapted to a different one with ease.</div><div><br /></div><div>Propel Peer classes come packed with the following method.</div><div><br /></div><div><code>BaseUserPeer::retrieveByPK($pk, $con=null);</code></div><div><br /></div><div>I would like to override it in the following way:</div><div><br /></div><div><code></code></div><code><div>public static function retrieveByPK($pk, $con = null)</div><div>{</div><div> $cacheKey = sprintf('user:id:%d', $pk);</div><div> $asArray = $memcache->get($cacheKey);</div><div> </div><div> if($asArray === null)</div><div> {</div><div> $obj = parent::retrieveByPK($pk, $con);</div><div> if($obj !== null)</div><div> {</div><div> $memcache->set($cacheKey, $obj->toArray(BasePeer::TYPE_FIELDNAME));</div><div> }</div><div> }</div><div> else</div><div> {</div><div> $obj = new User();</div><div> $obj->fromArray($asArray, BasePeer::TYPE_FIELDNAME);</div><div> }</div><div> </div><div> return $obj;</div><div>}</div></code><div></div><div><br /></div><div>Note that I've avoided the memcache connection code. I assume that there is a memcache class that abstracts the process. I'm also avoided the details of this class instantiation.</div><div><br /></div><div>As you can see from the code, I store the object in memcache as an associative array. I like to do so because I don't want to store a serialized version of the Propel object which will take more space. Also by using a native data type I can warm up the cache from -let's say- a batch script without the need of using Propel at all. Also, if a I have code that don't require symfony or Propel, I can still use the cached data. </div><div><br /></div><div>The <span class="Apple-style-span" style="font-weight: bold;">fromArray</span> and <span class="Apple-style-span" style="font-weight: bold;">toArray</span> methods are built inside propel objects as a convenient way of populating them, so there's no extra effort in our side to get their benefits. </div><div><br /></div><div>As key for the cache I'm using as prefix the name of the table, followed by a colon and then the primary key column name, a colon and the primary value. i. e.: <span class="Apple-style-span" style="font-weight: bold;">table_name:primary_key:value</span></div><div><br /></div><div>Then what is left for this to actually work is to override the User::save($con = null) method, so every time the database row is updated the changes will be reflected in the cache.</div><div><br /></div><div>I came up with the following code:</div><div><br /></div><div><code> </code></div><code><div>public function save($con = null)</div><div>{</div><div> $affectedRows = parent::save();</div><div> $memcache->set(sprintf('user:id:%d', $this->getId()), $this->toArray(BasePeer::TYPE_FIELDNAME));</div><div> return $affectedRows;</div><div>}</div></code><div></div><div><br /></div><div>There by updating the entry after it's saved to the database I used a traditional pattern with memcache to warm up the cache.</div><div><br /></div><div>So lets say that in a normal login form, the user will submit his <span class="Apple-style-span" style="font-weight: bold;">nickname</span> and <span class="Apple-style-span" style="font-weight: bold;">password</span> to be checked against the database. In this case we have to do a SELECT query</div><div>using the nickname and password as WHERE parameters. Instead of issuing a query, I would like to do the following inside UserPeer::retrieveByNickname($nickname).</div><div><br /></div><div><code></code></div><code><div>public static function retrieveByNickname($nickname)</div><div>{</div><div> $nicknameCacheKey = sprintf('user:nickname:%s', $nickname);</div><div> $userId = $memcache->get($nicknameCacheKey);</div><div> </div><div> if($userId !== null)</div><div> {</div><div> return UserPeer::retrieveByPK($userId);</div><div> }</div><div> else</div><div> {</div><div> $c = new Criteria();</div><div> $c->add(UserPeer::NICKNAME, $nickname);</div><div> $user = UserPeer::doSelectOne($c);</div><div> if($user !== null)</div><div> {</div><div> $memcache->set(sprintf('user:id:%d', $user->getId()), $user->toArray(BasePeer::TYPE_FIELDNAME));</div><div> $memcache->set($nicknameCacheKey, $user->getId());</div><div> }</div><div> </div><div> return $user;</div><div> }</div><div>}</div></code><div></div><div><br /></div><div>In the last example first I check if there is a memcache entry with the following key: <span class="Apple-style-span" style="font-weight: bold;">user:nickname:somenickname</span>. The value stored will be the user id which I assume</div><div>is the primary key of the table. If the user id is not null then I delegate the call to <span class="Apple-style-span" style="font-weight: bold;">UserPeer::retrieveByPK</span> to do the job. In the other case I fetch the user from</div><div>the database using the nickname as Criteria condition. If the record exists I store inside memcache the user data as an array and also I store the id using the</div><div>user:nickname:somenickname key. Now should be clear why in the previous example I used used:id:somenid as key.</div><div><br /></div><div>Some improvements to do in the <span class="Apple-style-span" style="font-weight: bold;">retrieveByPK</span> method and in the <span class="Apple-style-span" style="font-weight: bold;">save</span> method will be to also store the respective values for the user:nickname:somenickname key once</div><div>the object has been populated. In this way we increase the chances that retrieveByNickname will successfully hit the memcache.</div><div><br /></div><div>I hope this article results useful for you and thanks for reading.</div><div><br /></div><div>NOTE: I know that the code is pretty ugly, some code needs to be refactored out to it's own methods and maybe Propel classes are not the best place for</div><div>this caching logic to reside, but I think is a nice example to build upon.</div>Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-6669934595599937894.post-35382420488146088142009-03-03T06:49:00.000-08:002009-03-03T08:32:55.902-08:00Symfony Speed and Hello World Benchmarks<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">After reading some posts showing that my blah blah framework is way more fast than symfony for a Hello World application I decided to explain why: because symfony is extensible and can adapt to your needs. That’s easy to say you may think, in fact, every framework out there claims that. So what makes symfony so special?<br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">The following list names some of the features provided by symfony.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"></p><ul><li>Factories<br /></li><li>The Filter Chain <br /></li><li>The Configuration Cascade<br /></li><li>The Plugins System<br /></li><li>Controller adaptability<br /></li><li>View adaptability<br /></li></ul><p></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><span class="Apple-style-span" style="font-weight: bold;">Factories</span></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Since version 1.0 symfony provides a configuration file called <span class="Apple-style-span" style="font-weight: bold;">factories.yml</span>. This file affects the application core classes configuration. There you can override symfony default classes by your own ones. This means you can set up a custom Front Controller, Web Request, Cache classes, Session storage, etc. </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">But this come with an extra price: when a symfony application bootstraps, it reads the configuration file from the filesystem. If the yml file was parsed before, then it loads a PHP file -which can also be cached with APC-, if not, then it parses the yml file, stores the parsed file on the cache folder and loads the configuration from there.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Why should I need that flexibility you may ask? In a project I’m involved with we needed Memcached. This means that we overrode all of the symfony default cache mechanism by our own custom classes. How? Setting up our classes in an easy to read yml file. So for sure when you benchmark your Hello World application symfony will be slower.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><span class="Apple-style-span" style="font-weight: bold;">Filter Chain</span></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">One of the patterns from the <span class="Apple-style-span" style="font-weight: bold;">Core J2EE Patterns</span> book that impressed me the most is the <span class="Apple-style-span" style="font-weight: bold;">Intercepting Filter</span>. This patterns teach how to modify a request processing without the need to change Controllers or Model code. The idea is that in a configuration file you plug a class that will take care of pre or post processing the request. This classes are called filters. As an example, you can add a filter that checks if the user has the proper credentials to execute the action she wants. Another filter can cache the response, etc. </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Symfony has the <span class="Apple-style-span" style="font-weight: bold;">filters.yml</span> file which can be specified by application or by module. This means that we can set filters to be executed for the whole application, and then for specific modules -let’s say for Ajax actions-, we disable them. Does your framework provides this flexibility without resorting to some kind of monkey patching techniques? No? Well symfony does. Say hello to the <span class="Apple-style-span" style="font-weight: bold;">Filter Chain</span>. So for sure when you benchmark your Hello World application symfony will be slower. Because it adds flexibility to the process. You want to get rid of this behavior? Sure, set up a custom controller in the factories.yml file, and in your new class override the loadFilters method.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><span class="Apple-style-span" style="font-weight: bold;">Configuration Cascade</span></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">As explained in the <a href="http://www.symfony-project.org/book/1_0/05-Configuring-Symfony">configuration</a> chapter of the symfony book, symfony allows to modify it’s behavior through some yml files. So for example we have the <span class="Apple-style-span" style="font-weight: bold;">view.yml</span> that tells symfony which css and javascript files it should load for the current request. We can have an application <span class="Apple-style-span" style="font-weight: bold;">view.yml</span> configuration and then override the settings per module. When symfony process the request it checks all this files, that’s why in the Hello World benchmark it‘s slow.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><span class="Apple-style-span" style="font-weight: bold;">Plugins System</span></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Symfony has a very powerful and easy to use Plugin System. It’s more than <span class="Apple-style-span" style="font-weight: bold;">400 plugins</span> with a set of <span class="Apple-style-span" style="font-weight: bold;">200+ developers</span> speaks by itself of it success. </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">The plugins can contain modules of their own and also a config.php file, similar to the project config.php file or the module config.php. When symfony process a request it checks for the settings in those files, this means that they will be read from disk. So in a plugin we can provide a custom logger that is fired up when bootstrapping the application. The Plugin user doesn’t need to care how the logger will be activated, she just now that it will work. </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">The same applies for plugin modules. How do you think that symfony knows that certain module/action should be called from a plugin? If you enable the plugin module on the settings.yml file symfony will check inside the plugin module to see if the requested action should be executed there. </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><span class="Apple-style-span" style="font-weight: bold;">Controller adaptability</span></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">For each action that the user call symfony will execute a page controller. Inside our modules symfony let us use a generic myModuleActions class that will extend sfActions or one specific to the action requested by the user, that as an example could be called indexAction and will extend the sfAction class. When a request is processed symfony first checks for the existence of the later. If it doesn’t exists then it tries to load the generic action for that module. Of course you don’t need this kind of flexibility for Hello World apps.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><span class="Apple-style-span" style="font-weight: bold;">View Adaptability</span></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">For rendering the response symfony uses by default the <span class="Apple-style-span" style="font-weight: bold;">sfPHPView</span> class. If certain module in your application requires a different view, then there are at least <a href="http://obvioushints.blogspot.com/2009/01/custom-views-in-symfony-10.html">three</a> ways to accomplish this as explained here.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><span class="Apple-style-span" style="font-weight: bold;">Conclusion</span></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Symfony is a Professional Web Application Framework built to cope with real world needs. In a large project with more than a simple salutation feature sooner or later you will need the flexibility provided by the framework. This will save you time and will prevent headaches, because when you have built a whole system with a framework and the business needs start to push in a direction where you have to extend the framework you will thank yourself for having choose symfony at first.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">In case that your client requires a Hello World! application, then you can use the following <span class="Apple-style-span" style="font-weight: bold;">hyper fast framework code</span>: <span class="Apple-style-span" style="font-weight: bold;">die(“Hello World!”)</span> ;-)</span></p>Unknownnoreply@blogger.com10tag:blogger.com,1999:blog-6669934595599937894.post-63829026325254699182009-01-28T07:29:00.001-08:002009-01-28T07:43:21.578-08:00Integrating Facebook Hive with Symfony<p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">A symfony feature that has been really helpful while working on projects with it is the <a href="http://www.symfony-project.org/book/1_0/16-Application-Management-Tools#Symfony%20Logs">debugging</a> capabilities of the framework. Almost every day I see my self going to the command line an doing a <code>tail -f log/frontend_dev.log </code> in my symfony project. This helps me see what is going on behind the scenes, spot bugs, find places where I can improve the code, etc. If something goes wrong there is always a developer on the team that shouts “check the symfony logs”, showing that they had became an essential tool for development. <br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">But not everything shines under the sun. Sometimes it happens that we would like to have a tool that allows us to filter the logs according to specific criteria. A tool that goes beyond a simple <code>cat logs/frontend_dev.log | grep SELECT</code>. We wanted to perform some analysis on the website usage, basically to help us improve it performance.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">A colleague talked about <a href="http://www.facebook.com/note.php?note_id=32008268919">Facebook Scribe</a>, that was not actually related with this dream tool but later lead me to learn about the existence of the <a href="http://wiki.apache.org/hadoop/Hive">Apache Hive</a> project (which was started by Facebook).</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">The Hive project allows to load a logs file and then filter it with SQL like commands. -Hive is more than this raw description, you can read more about it <a href="http://wiki.apache.org/hadoop/Hive/GettingStarted">here</a>.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">My idea was to adapt the file logging format of symfony to make easy to import those files inside a Hive database. Because Hive support table partitioning by date, it should be easier to load the data from the logs and then perform the analysis with the SQL like syntax provided by Hive.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">After some fights with <span class="Apple-style-span" style="font-weight: bold;">Ant</span> and <span class="Apple-style-span" style="font-weight: bold;">Java 1.6</span> It was possible for me to get Hive running in my Mac. Then I just created a shameless copy of the <span class="Apple-style-span" style="font-weight: bold;">sfFileLogger</span> and renamed it to <span class="Apple-style-span" style="font-weight: bold;">sfHiveLogger</span>. I did small changes here and there and got it ready to log in a format that it’s easy to load later into Hive. I browsed through my testing symfony project to generate some logs and then I moved to the command line to start the fun with Hive.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">There I created a table to hold the logs with the following command:</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">CREATE TABLE sflogs(</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> logTime STRING, </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> priorityName STRING,</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> message STRING</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">) </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> COMMENT 'This is the sflogs table' </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> PARTITIONED BY(dt STRING) </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> ROW FORMAT DELIMITED</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> FIELDS TERMINATED BY '\011' </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> LINES TERMINATED BY '\012';</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"></code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Everything was working smoothly. The next step was to load the data from the logs file into the sflogs table to start issuing queries to it. I did it with this command:</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">LOAD DATA LOCAL INPATH '/path/to/myproject/log/frontend_dev.log.2009-01-28' INTO TABLE sflogs PARTITION (dt='2009-01-28');</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"></code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">With the data loaded I started to issue some commands against the Hive console like:</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">SELECT * FROM sflogs WHERE message LIKE "{sfRequest}%";</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">SELECT DISTINCT priorityname FROM sflogs;</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">SELECT COUNT(1) FROM sflogs;</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"></code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica">The results were similar as when we work on a mysql client which was awesome. What amazed me the most was how with some easy changes it's possible to adapt symfony to our needs. But which were this changes? Here they are:</p><p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px">First we need to enable the sfHiveLogger in the logging.yml file of your symfony project under the <span class="Apple-style-span" style="font-weight: bold;">sf_file_debug</span> entry<br /><span style="letter-spacing: 0.0px"></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Then do the shameless copy of the symfony class to your lib folder and rename it to <span class="Apple-style-span" style="font-weight: bold;">sfHiveLogger</span>.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Then change the code of the <span class="Apple-style-span" style="font-weight: bold;">initialize</span> method to look like this:</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> if (!isset($options['file']))</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> {</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> throw new sfConfigurationException('File option is mandatory for a file logger');</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> }</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> $dir = dirname($options['file']);</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> if (!is_dir($dir))</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> {</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> mkdir($dir, 0777, 1);</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> }</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"> </span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> $logFileName = $options['file'] . '.' . date('Y-m-d');</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> $fileExists = file_exists($logFileName);</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> if (!is_writable($dir) || ($fileExists && !is_writable($logFileName)))</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> {</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> throw new sfFileException(sprintf('Unable to open the log file "%s" for writing', $logFileName));</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> }</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> $this->fp = fopen($logFileName, 'a');</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> if (!$fileExists)</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> {</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> chmod($logFileName, 0666);</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"> }</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"></code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Basically the change there is to append the current date in “Y-m-d-” format at the end of the logs file name. This will make easier to import the logs into Hive -only if you want them partitioned by date-.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Then on the log method change the line with:</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">$line = sprintf("%s %s [%s] %s%s", strftime('%b %d %H:%M:%S'), 'symfony', $priorityName, $message, DIRECTORY_SEPARATOR == '\\' ? "\r\n" : "\n");</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"></code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">to:</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">$line = sprintf("%s\t%s\t%s%s", strftime('%b %d %H:%M:%S'), $priorityName, $message, "\n");</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"></code></span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Here what we do is to apply a little formating there. The most important part is to have the <span class="Apple-style-span" style="font-weight: bold;">tab</span> and then <span class="Apple-style-span" style="font-weight: bold;">new line</span> characters as delimiters because this was what we specified in the CREATE TABLE command above.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">With this easy steps we can have a Hive enabled log file. If you want to learn more about Hive and the supported commands and the theory behind it please refer to the <a href="http://wiki.apache.org/hadoop/Hive">wiki</a>.</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px"><span class="Apple-style-span" style="font-weight: bold;">Conclusion</span>:</span></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; min-height: 14.0px"><span style="letter-spacing: 0.0px"></span><br /></p> <p style="margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica"><span style="letter-spacing: 0.0px">Besides that I’m still comparing Hive with other solutions to parse and analyze the logs, I think that this tool has a lot of potential to help debugging and profiling symfony applications. If we polish the log format and refine the table structure, then it’s shouldn’t be hard to setup some cronjobs that generate reports of the website usage, improving the usability of the symfony logs.</span></p>Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6669934595599937894.post-28921220467540265322009-01-26T07:44:00.000-08:002009-01-26T08:07:13.499-08:00Custom Views in Symfony 1.0<div>After seeing <a href="http://symfony.uservoice.com/pages/symfony/suggestions/109512-add-abitity-to-use-custom-myphpview-class">this</a> feature request for symfony 1.3 I decided to write a tutorial explaining how to use custom views with symfony 1.0. Yes symfony 1.0 has this feature, a little bit hide inside, but is there since the <a href="http://www.symfony-project.org/book/1_0/06-Inside-the-Controller-Layer#Module%20Configuration">ol’ good days</a>.<br /></div><div><br /></div><div>So, there are three ways to have a custom View class for your actions. To show how this work, we will need a project with a <span class="Apple-style-span" style="font-weight: bold;">module</span> called <span class="Apple-style-span" style="font-weight: bold;">example</span>. </div><div><br /></div><div><span class="Apple-style-span" style=""><span class="Apple-style-span" style="font-weight: bold;">First example:</span></span></div><div><br /></div><div>After you have your module ready add an action called FirstExample which will have this code: </div><div><br /></div><div><code>public function executeFirstExample() {}</code></div><div><br /></div><div>Then on the <span class="Apple-style-span" style="font-weight: bold;">example</span> module add a folder called <span class="Apple-style-span" style="font-weight: bold;">views</span>. Inside it we will add our custom view class. The name will be <span class="Apple-style-span" style="font-weight: bold;">firstExampleSuccessView</span>. Create a <span class="Apple-style-span" style="font-weight: bold;">firstExampleSuccessView.class.php</span> file with the following code inside:</div><div><br /></div><div><code><div>class firstExampleSuccessView extends sfPHPView</div><div>{</div><div> </div><div>}</div></code><div></div><div><br /></div><div>As you can see, it extends the <span class="Apple-style-span" style="font-weight: bold;">sfPHPView</span> provided by symfony, to get all the functionality inside. Please note that custom views in symfony need to extend <span class="Apple-style-span" style="font-weight: bold;">sfView</span>, which is the parent class of sfPHPView. </div><div><br /></div><div>At this point I’m going to tell you that we can customize a lot of features of the symfony views. For the tutorial I will just show you how to have new shortcut variables inside the templates like the <span class="Apple-style-span" style="font-weight: bold;">$sf_request</span> or <span class="Apple-style-span" style="font-weight: bold;">$sf_user</span> already provided by symfony. Feel free to comment about your experience extending the sfView.</div><div><br /></div><div>To add a new shortcut to the template we need to extend the following method:</div><div><span class="Apple-style-span" style="font-weight: bold;">sfPHPView::getGlobalVars()</span>. We will add a new shortcut smartly called <span class="Apple-style-span" style="font-weight: bold;">my_view_data</span>. with a dummy array inside. This will be our code:</div><div><br /></div><div><code></code></div><code><div>protected function getGlobalVars()</div><div>{</div><div> $context = $this->getContext();</div><div><br /></div><div> $shortcuts = array(<br /></div><div> 'sf_context' => $context,</div><div> 'sf_params' => $context->getRequest()-></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>getParameterHolder(),</div><div> 'sf_request' => $context->getRequest(),</div><div> 'sf_user' => $context->getUser(),</div><div> 'sf_view' => $this,</div><div> 'my_view_data' => array('foo' => 'this data came from firstExampleSuccessView')</div><div> );</div><div><br /></div><div> if (sfConfig::get('sf_use_flash'))</div><div> {</div><div> $sf_flash = new sfParameterHolder();</div><div> $sf_flash->add($context->getUser()-></div><div><span class="Apple-tab-span" style="white-space:pre"> </span> getAttributeHolder()-></div><div><span class="Apple-tab-span" style="white-space:pre"> </span> getAll('symfony/flash'));</div><div> $shortcuts['sf_flash'] = $sf_flash;</div><div> }</div><div><br /></div><div> return $shortcuts;</div><div>}</div></code><div></div><div><br /></div><div>As you can see there is a new <span class="Apple-style-span" style="font-weight: bold;">my_view_data</span> key with a one element array.</div><div><br /></div><div>To see this in action we add in the <span class="Apple-style-span" style="font-weight: bold;">example/templates</span> folder a file called <span class="Apple-style-span" style="font-weight: bold;">firstExampleSuccess.php</span> with the following content:</div><div><br /></div><div><div><code></code></div><code><div><h1>First Example View:</h1></div><div><?php var_dump($my_view_data); ?></div></code><div></div></div><div><br /></div><div>Clear the cache and point your browser to http://yourapp.com/example/firstExample</div><div><br /></div><div>There you will see the output of our new shortcut.</div><div><br /></div><div>As long as our action returns <span class="Apple-style-span" style="font-weight: bold;">sfView::SUCCESS</span> it will use our new custom view class. Who said that symfony had no custom views? :-)</div><div><br /></div><div>Remember to follow the naming convention of: <<span class="Apple-style-span" style="font-weight: bold;">actionName</span>><<span class="Apple-style-span" style="font-weight: bold;">viewName</span>><span class="Apple-style-span" style="font-weight: bold;">View</span>. In our example we had firstExampleSuccessView. firstExample is the action, Success is on of the possible values that our actions can return, View is the default suffix. </div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">Second Example</span></div><div><br /></div><div>In this case we will create a class called <span class="Apple-style-span" style="font-weight: bold;">secondExampleView</span> inside a file <span class="Apple-style-span" style="font-weight: bold;">secondExampleView.class.php</span> located in the <span class="Apple-style-span" style="font-weight: bold;">lib</span> folder of our application. The contents of this class will be almost the same that four our first view. We will change the class name to secondExampleView and the my_view_data entry will be declared like this:</div><div><br /></div><div><code>'my_view_data' => array('foo' => 'this data came from secondExampleView')</code></div><div><br /></div><div>Then we create a new action for our module with the following code:</div><div><br /></div><div><code>public function executeSecondExample(){}</code></div><div><br /></div><div>Then we add the template <span class="Apple-style-span" style="font-weight: bold;">secondExampleSuccess.php</span> with this code inside:</div><div><br /></div><div><div><code></code></div><code><div><h1>Second Example View:</h1></div><div><?php var_dump($my_view_data); ?></div></code><div></div><div><br /></div></div><div>Now, how do we use our new view with this action? In case we don’t have one, we add a <span class="Apple-style-span" style="font-weight: bold;">module.yml</span> file in the <span class="Apple-style-span" style="font-weight: bold;">config</span> folder of the example module. There we add the following content:</div><div><br /></div><div><code></code></div><code><div>all:</div><div> view_class: secondExample</div></code><div></div><div><br /></div><div>This will tell symfony to use our custom view class for this module. As you can see here, we can specify different views per environment :-) Yes, symfony is configurable :-) </div><div><br /></div><div>If we point our browsers to the secondExample action we will see the contests of our shortcut.</div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">Third Example</span>:</div><div><br /></div><div>For this example we will create a class called <span class="Apple-style-span" style="font-weight: bold;">thirdExampleView</span> inside the <span class="Apple-style-span" style="font-weight: bold;">lib</span> folder of our application. As we did with our secondExampleView we will have the same code, except for the class name and the my_view_data shortcut that will have inside the following: </div><div><br /></div><div><code></code></div><code><div>array('foo' => 'this data came from thirdExampleView')</div></code><div></div><div><br /></div><div>We add a template called <span class="Apple-style-span" style="font-weight: bold;">thirdExampleSuccess.php</span> with this content:</div><div><div><br /></div><div><code></code></div><code><div><h1>First Example View:</h1></div><div><?php var_dump($my_view_data); ?></div></code><div></div></div><div><br /></div><div>and we create an action like this:</div><div><br /></div><div><code></code></div><code><div>public function executeThirdExample()</div><div>{</div><div> $module = $this->getRequestParameter('module');</div><div> $action = $this->getRequestParameter('action');</div><div> $this->getRequest()->setAttribute($module.'_'.$action.'_view_name', 'thirdExample', 'symfony/action/view');</div><div>}</div></code><div></div><div><br /></div><div>What we do here is setting in the <span class="Apple-style-span" style="font-weight: bold;">'symfony/action/view'</span> namespace of the <span class="Apple-style-span" style="font-weight: bold;">sfRequest</span><span class="Apple-style-span" style="font-weight: bold;"> attribute</span> <span class="Apple-style-span" style="font-weight: bold;">holder</span> a value with our view class <span class="Apple-style-span" style="font-weight: bold;">without the View</span> <span class="Apple-style-span" style="font-weight: bold;">suffix</span>. The key of the attribute should be composed by <<span class="Apple-style-span" style="font-weight: bold;">module_name</span>>_<<span class="Apple-style-span" style="font-weight: bold;">action_name</span>><span class="Apple-style-span" style="font-weight: bold;">_view_name</span> as you can see from the code above.</div><div><br /></div><div>Then we can point our browsers to the thirdExample action to see the results.</div><div><br /></div><div>With this three ways we can set up your own views and customize the rendering process of our applications. </div><div><br /></div><div>As final note I want to add how symfony knows which view class to pick.</div><div><br /></div><div><ul><li>It checks inside the module/views folder to see if it can find our custom class as explained on the first example.<br /></li><li>If there is no user class there, it will check if we have set up one in the sfRequest as explained on example three.<br /></li><li>Then it will try to see if we defined a view in the module.yml file.<br /></li><li>As last resort it will load the default sfPHPView class.<br /></li></ul></div><div><br /></div><div>Thanks for reading this far and I hope this will help you on your projects.</div><div><br /></div></div>Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-6669934595599937894.post-60213078672451698122009-01-15T06:25:00.000-08:002009-01-15T07:25:00.317-08:00Firebug Framework: CSS helper functionsIn a <a href="http://obvioushints.blogspot.com/2008/10/firebug-framework-whats-inside.html">previous</a> post I started talking about the functionality that is already implemented in Firebug. While I was doing some reworking on <a href="http://www.firesymfony.org/">FireSymfony</a> I had the chance to use some of the CSS methods that are part of the <a href="http://www.getfirebug.com/">Firebug</a> <code>lib.js</code> source file:<div><br /></div><div><div><span class="Apple-style-span" style="font-weight: bold;"><code>hasClass = function(node, name)</code></span><br /></div><div><br /></div><div>This function expects a node, that for example can be obtained with Firebug "$()" function, and the CSS class name that we are searching for. Returns boolean as expected.</div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"><code>setClass = function(node, name)</code></span></div><div><br /></div><div>Adds the CSS class provided by <code>name</code> to the <code>node</code></div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"><code>removeClass = function(node, name)</code></span></div><div><br /></div><div>Removes the class specified by <code>name</code> from <code>node</code></div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"><code>toggleClass = function(elt, name)</code></span><br /></div><div><br /></div><div>If <code>elt</code> has class <code>name</code> then it will remove it from the element, otherwise it will add the class to the element.</div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"><code>setClassTimed = function(elt, name, context, timeout)</code></span></div><div><br /></div><div>This will add the class <code>name</code> to the element. When the milliseconds timeout has expired, the class will be removed from the element and the timeout will be cleared. The timeout is added to the provided Firebug <code>context</code></div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"><code>cancelClassTimed = function(elt, name, context)</code> </span></div><div><br /></div><div>This method will remove a class added with the previous method and will clear the timeout.</div><div><br /></div><div>All this methods can be accessed from inside our Firebug extensions if we declared them as explained <a href="http://www.softwareishard.com/blog/firebug-tutorial/extending-firebug-hello-world-part-i/">here</a> by Jan Odvarko. Another way is calling them like this: <code>FBL.methodName(...)</code></div><div><br /></div><div>There are more utilities inside the Firebug libraries that can help us while we develop our custom extensions, so we don't have to reinvent the wheel. I've plans of keep documenting the Firebug code, so stay tuned.</div><div><br /></div><div><br /></div></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6669934595599937894.post-55530991761568074502008-12-19T08:21:00.000-08:002008-12-19T09:37:24.997-08:00Extending WebKit Web Inspector with Fun!<div>After being astonished by the code that <a href="http://www.25lines.com/finalists/0812/043.txt">won</a> the ActionScript contest of <a href="http://www.25lines.com/">25lines.com</a> I decided to see if it was possible to port it to Javascript. <div><br /></div><div>I created an html page to hold the game and then worked on the port. After some time, I had the game working on Firefox and Safari, which was more than enough. Then the crazy idea came into my mind... </div><div><br /></div><div>Why not to show how to extend WebKit Web Inspector with this game? -I can do a hello world example if you want.</div><div><br /></div><div>So, to show you how to add a custom Panel to the Web Inspector, in a similar way in how Firebug allows to be extended, we will add the game. </div><div><br /></div><div>Here you can see an screenshot of the final project inside Web Inspector:</div><div><br /></div></div><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXEnJZhV1Jqr2BSbWhjMb7ReXeqO6nia8AASSzbL6Da4Ai-B6ApXBu3NQuDhkDgj_EdJYmQmBQvnOErTTk7gY5kZo5pd1iSFFfDHNDSawEcUJdKj1w8CIu320zH6eBe0F0JdUJ-lQ-rZE/s1600-h/fun_panel.jpg"><img style="cursor:pointer; cursor:hand;width: 400px; height: 225px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXEnJZhV1Jqr2BSbWhjMb7ReXeqO6nia8AASSzbL6Da4Ai-B6ApXBu3NQuDhkDgj_EdJYmQmBQvnOErTTk7gY5kZo5pd1iSFFfDHNDSawEcUJdKj1w8CIu320zH6eBe0F0JdUJ-lQ-rZE/s400/fun_panel.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_5281553336184627810" /></a><div><div><div><br /></div></div><div>WebKit Web Inspector is easy to extend. First we need to find where it's installed. If you don't have it already, you can get it from <a href="http://webkit.org/">here</a>. In my Mac it resides in the following folder: <br /></div><div><br /></div><div><code>/Applications/WebKit.app/Contents/Frameworks/10.5/WebCore.framework/Versions/A/Resources/inspector</code></div><div><br /></div><div>There you will find the following files: <code>inspector.html</code> and <code>inspector.js</code></div><div><br /></div><div>Those files are the ones that we will modify to add our custom Panel with the game, but before we need to grab the code from the google project <a href="http://code.google.com/p/smiliebounce25forjs/">page</a> -you will need to use svn-.</div><div><br /></div><div>After you get the code, you will find two folders, inside <code>jsport</code> you can find the working HTML/Javascript example. Inside <code>inspector</code> there are the files for the panel. Copy <code>game.css</code> and <code>game.js</code> to the folder where you have installed Web Inspector. Then edit <code>inspector.html </code>adding the following lines below the <code>inspector.js</code> include line:</div><div><br /></div><div><code></code></div><code><div> <script type="text/javascript" src="game.js"></script></div></code><div><code> <link rel="stylesheet" type="text/css" href="game.css"></code></div><div><br /></div><div>Then open <code>inspector.js</code> and replace the following code:</div><div><br /></div><div><code></code></div><code><div> this.panels = {</div><div> elements: new WebInspector.ElementsPanel(),</div><div> resources: new WebInspector.ResourcesPanel(),</div><div> scripts: new WebInspector.ScriptsPanel(),</div><div> profiles: new WebInspector.ProfilesPanel(),</div><div> databases: new WebInspector.DatabasesPanel()</div></code><div><code> };</code></div><div><br /></div><div>with the following one:</div><div><br /></div><div><code></code></div><code><div> this.panels = {</div><div> elements: new WebInspector.ElementsPanel(),</div><div> resources: new WebInspector.ResourcesPanel(),</div><div> scripts: new WebInspector.ScriptsPanel(),</div><div> profiles: new WebInspector.ProfilesPanel(),</div><div> databases: new WebInspector.DatabasesPanel(),</div><div> game: new WebInspector.GamePanel()</div></code><div><code> };</code><br /></div><div><br /></div><div>What we did here was add our <span class="Apple-style-span" style="font-weight: bold;">GamePanel</span> to the hash of panels that Web Inspector will initialize.</div><div><br /></div><div>Then If you see inside <code>game.js</code> you will find the following -I omitted some code for brevity-:</div><div><br /></div><div><code></code></div><code><div>WebInspector.GamePanel = function()</div><div>{</div><div>WebInspector.Panel.call(this);</div><div>...<br /></div><div>};</div><div><br /></div><div>WebInspector.GamePanel.prototype = {</div><div> toolbarItemClass: "scripts",</div><div><br /></div><div> get toolbarItemLabel()</div><div> {</div><div> ...</div><div> },</div><div><br /></div><div> show: function()</div><div> {</div><div> WebInspector.Panel.prototype.show.call(this);</div><div> </div><div> ...</div><div> },</div><div><br /></div><div> hide: function()</div><div> {</div><div> WebInspector.Panel.prototype.hide.call(this);</div><div> }</div><div>};</div><div><br /></div></code><div><code>WebInspector.GamePanel.prototype.__proto__ = WebInspector.Panel.prototype;</code><br /></div><div><br /></div><div>Those are the three basic parts of our Panel. First we declare it. Then we implement the interface of <code>WebInspector.Panel</code> and at the end we declare that our panel extends <code>WebInspector.Panel</code></div><div><br /></div><div>In the first part what we do is add a container for the game as a <span class="Apple-style-span" style="font-weight: bold;">pre </span>tag. This element is attached to <code>this.element</code> which is the base element of a <code>WebPanel.</code> By setting the id <span class="Apple-style-span" style="font-weight: bold;">gameArea, </span>we style the <span class="Apple-style-span" style="font-weight: bold;">pre </span>with the styles defined in <code>game.css</code></div><div><br /></div><div>Then on the second part I set the default css class for our panel as <span class="Apple-style-span" style="font-weight: bold;">scripts</span>. In this way, our button on the Web Inspector toolbar will have the same icon as the <span class="Apple-style-span" style="font-weight: bold;">Scripts</span> panel -if you want you can add a new style and define there a background image as you will do with CSS in a normal web page-.</div><div><br /></div><div>For <code>get toolbarItemLabel()</code> we return <span class="Apple-style-span" style="font-weight: bold;">Fun!</span>. This will be the label for our Panel in the toolbar.</div><div><br /></div><div>Then we implement <code>show()</code> where we draw the game. Here we can add the code that will initialize the contents of our Panel. Inside <code>hide()</code> We should implement all the logic to be run when the user switch to another panel, like removing unused objects, etc.</div><div><br /></div><div>After we have everything in place. We restart WebKit, open the Web Inspector and enjoy our game.</div><div><br /></div><div>NOTE 1: AFAIK, this technique should work for Google Chrome Web Inspector. The last time I checked the code, it was using the same code of WebKit.</div><div><br /></div><div>NOTE 2: The drawback of this kind of extensions is that the next time you upgrade the browser, it will erase your Panel. Maybe in the future WebKit allows a more <span class="Apple-style-span" style="font-style: italic;">plug and play</span> way of doing it. In <a href="http://obvioushints.blogspot.com/2008/10/firesymfony-to-support-webkit.html">this</a> previous post I show some screenshots of <a href="https://addons.mozilla.org/en-US/firefox/addon/9096/">FireSymfony</a> working inside Web Inspector.</div><div><br /></div><div>NOTE 3: The original author of the ActionScript game is Marius Heil.<br /></div></div>Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-6669934595599937894.post-46866863558643283802008-12-16T04:05:00.000-08:002008-12-16T16:43:20.427-08:00Symfony Design Patterns<div>Much <a href="http://blog.mikeseth.com/index.php?/archives/4-ActiveRecord-sucks,-but-Kore-Nordmann-is-wrong.html">has</a> <a href="http://pookey.co.uk/blog/archives/43-phplondon08-the-crazy-guy-mail.html">been</a> <a href="http://blog.astrumfutura.com/archives/373-The-M-in-MVC-Why-Models-are-Misunderstood-and-Unappreciated.html">said</a> this days about how modern web frameworks interpret the MVC architectural pattern. I'm my case those articles made me rethink how I use a framework, specially Symfony.</div><div><br /></div><div>This lead me to start a study on which patterns come into play while we develop a Symfony application. So while adding new features or refactoring existing ones I will know which class is in charge off doing the job.</div><div><br /></div><div>Some of the patterns involved in Symfony are:</div><div><ul><li>Front Controller<br /></li><li>Command<br /></li><li>Intercepting Filter<br /></li><li>Context Object<br /></li><li>Two Step View<br /></li><li>Helper Object or View Helper<br /></li><li>Table Data Gateway (i.e.: ArticlePeer.php)<br /></li><li>Row Data Gateway (i.e.: Article.php)<br /></li><li>Active Record<br /></li><li>Single Table Inheritance<br /></li></ul></div><div>How do they interoperate? The Front Controller act as an entry point for our application. After it processes the request it will choose which Command should handle the request, in our case, one of our defined actions, that's why we have the execute prefix as a convention for the action</div><div>names. </div><div><br /></div><div>While the request is being processed Symfony filter chain will launch modifying the results accordingly. The idea of the Intercepting Filter pattern is to be possible to pre and post modify the request and the response without the need of change existing code. What's is a nice feature in Symfony is that we can add them directly on the filters.yml file making really convenient to use. Also we can specify filters that only apply for certain modules, making the process really flexible.</div><div><br /></div><div>On the View side Symfony uses the Two Step View, so generally the result of our request will be decorated with a common layout to add consistency to our website. Here we can also use helpers, as they are explained in the Symfony documentation or add Helper classes. The Helpers classes can retrieve data from the Model, format data, or anything our application needs to work on the View side. With them we make our templates clear and free off script code, so it's easy to non developers to work on them.</div><div><br /></div><div>Also Symfony allows to use specific view classes by module, adding extra flexibility in the way we can handle the response, let's say, to return JSON data.</div><div><br /></div><div>As a way to keep a reference to the context, Symfony implement the -some guess please- Context Object pattern, which I won't talk about here.</div><div><br /></div><div>Then we have the DAL, which in my case is powered by Propel. This ORM implements the Table Data Gateway and the Row Data Gateway patterns. As they name implies they act as gateways to our tables and rows, but because generally we add some domain logic inside them they start to act as Active Records.</div><div><br /></div><div>A cool feature of Propel is that it implements the Single Table Inheritance pattern. This pattern is useful when we have a table for publications where we store Magazines and Books. If our table has a field acting as the type of the row, and in our application we treat them differently, whether if they are books or magazines, then this pattern let us work with Publication, Book and Magazine classes.</div><div><br /></div><div>After this research I came with the following conclusion:</div><div><ul><li>Use the controllers to handle user input data and choose which view to display.<br /></li><li>If there, remove all business logic from the controllers.<br /></li><li>Let the Model do it's job with the provided data. <br /></li><li>When handling data to the model, do it as an array of normalized data -or proper objects pertinent to our application-, the model shouldn't know about the request.<br /></li><li>Let helpers or Helper Objects handle the data formating, avoiding script code inside templates.<br /></li><li>Use Propel as a DAL and refactor towards a Model. This means to remove complex business logic from the Propel classes.</li></ul><div>What helped my during this research where the following books, which I highly recommend.</div><div><div><ul><li><a href="http://www.amazon.com/Enterprise-Application-Architecture-Addison-Wesley-Signature/dp/0321127420/">Patterns of Enterprise Application Architecture</a>.<br /></li><li><a href="http://www.amazon.com/Core-J2EE-Patterns-Practices-Strategies/dp/0131422464/">Core J2EE Patterns: Best Practices and Design Strategies.</a><br /></li></ul></div></div></div>Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-6669934595599937894.post-14595563763065719572008-11-20T03:47:00.000-08:002008-11-20T03:51:31.578-08:00New tncPropel12LoadbalancerPlugin released by TheNetCircle.comAt the <a href="http://www.thenetcircle.com">company</a> I work for we released a new symfony plugin. This time is to add a load balancing feature to Propel 1.2 and symfony 1.0. <div><br /></div><div>Check the article about it <a href="http://www.thenetcircle.com/blog/2008/11/20/database-load-balancing-with-propel-12/">here</a> and the plugin page <a href="http://www.symfony-project.org/plugins/tncPropel12LoadbalancerPlugin">here</a>.</div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6669934595599937894.post-13341028966611762462008-10-31T07:25:00.000-07:002008-10-31T07:36:40.343-07:00The Propel ORM - At php|architectI'm glad to announce that my article about Propel, one of the most known ORM solutions for PHP has been published in the <a href="http://www.phparch.com/c/magazine/issue/84">current</a> edition of php|architect. So all of you Propel and symfony lovers take a look at this edition for some insight on the usage of the ORM. Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6669934595599937894.post-84473974450365452152008-10-22T04:57:00.000-07:002008-10-22T05:05:59.453-07:00Gmail doesn't support Safari?<div><a href="http://obvioushints.blogspot.com/2008/10/msn-live-advices-on-browsers.html">Last time</a> my wife had a problem using MSN Live on Safari, but this time is Gmail who refuses to work with Safari asking for a supported browser. Here is the screenshot.</div><div><br /></div><div>. <a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_qJeWLZjlluBxLFQ0RFo07M2Iz9Bs0zIpF3br9gSNSRlNOUvrklFZfNRxOHryLV75RhJiY9Tm0wmecjzI2iBp_2lieTRIPsOaePCOEYOoKly7tnIbwU6j0jiIKUrgfovCyC3qXVaX2RI/s1600-h/GmailSafariPuHao.jpg"><img style="cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_qJeWLZjlluBxLFQ0RFo07M2Iz9Bs0zIpF3br9gSNSRlNOUvrklFZfNRxOHryLV75RhJiY9Tm0wmecjzI2iBp_2lieTRIPsOaePCOEYOoKly7tnIbwU6j0jiIKUrgfovCyC3qXVaX2RI/s400/GmailSafariPuHao.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_5259947488910990578" /></a><br /></div>Unknownnoreply@blogger.com2