Monday, November 16, 2009

ftp using Groovy and Grape

People always look at me in disbelieve when I say I use groovy to setup a bunch of cron scripts to ftp and process millions of records daily. Then they are even more surprised when I said I can set it up in a matter of minutes. Here is how I do it.
import org.apache.commons.net.ftp.FTPClient

start()

@Grab(group='commons-net', module='commons-net', version='2.0')
def start()
{
  def ftpClient = new FTPClient()
  ftpClient.connect("ftp server address here")
  ftpClient.enterLocalPassiveMode()
  println(ftpClient.replyString)
  ftpClient.login("username","password")
  println(ftpClient.replyString)
  ftpClient.changeWorkingDirectory("directory the file is in")
  println(ftpClient.replyString)
  ftpClient.fileType=(FTPClient.BINARY_FILE_TYPE)
  println(ftpClient.replyString)

  def incomingFile = new File("localfilename")
  incomingFile.withOutputStream { ostream ->
    ftpClient.retrieveFile("remotefilename", ostream )
  }
  println(ftpClient.replyString);
  ftpClient.disconnect()
}
I use Grab so I don't have to worry about dependencies. At each step after issuing a command, I then do a "println(ftpClient.replyString)" to see the output for diagnostic purposes. I'm also using "enterLocalPassiveMode()" to use ftp passive mode. I wrapped it in start() to work around the @Grab in Groovy 1.6 must be attached to something. You won't need to do that if you are using Groovy 1.7. I also set the transfer mode to explicitly binary because most of the times I'm dealing with zip files. Then I open a local file, use Groovy's with block to manage the download. Then disconnect. Groovy makes this so simple it takes me longer to explain it than writing the code, which is why I post it here instead of keep emailing this to everyone who asked. :) UPDATE: Guillaume Laforge, the master of Groovy himself suggested I use with.{} block to avoid repeating the prefix over and over again. He posted it here. It's a great suggestion. I must admit to this date, I often forget about with.{}. I've reproduced his code below.
@Grab(group='commons-net', module='commons-net', version='2.0')
import org.apache.commons.net.ftp.FTPClient

new FTPClient().with {
    connect "ftp server address here"
    enterLocalPassiveMode()
    login "username", "password"
    changeWorkingDirectory "directory the file is in"
    ftpClient.fileType = FTPClient.BINARY_FILE_TYPE
    def incomingFile = new File("localfilename")
    incomingFile.withOutputStream { ostream -> retrieveFile "remotefilename", ostream }
    disconnect()
}

Tuesday, November 10, 2009

Ninety-Nine Problems In Clojure Part 1 1-15

The original Ninety-Nine Prolog Problems was written by Werner Hett at the Berne University of Applied Sciences in Berne, Switzerland. Phil Gold then adapted it to Ninety-Nine Scala Problems. I had fun reading them. It has inspired me to do a series of Ninety-Nine Problems in different languages. I'm starting with Clojure. I'm taking a more practical approach. I use built-in library whenever possible. I use clojure-contrib as well. Because on a day to day basis, this is what we all do. They exist for a reason after all. If you feel that's cheating. Feel free to write your own. :) The difficulty ranking was for Prolog. The original text says
The problems have different levels of difficulty. Those marked with a single asterisk (*) are easy. If you have successfully solved the preceeding problems you should be able to solve them within a few (say 15) minutes. Problems marked with two asterisks (**) are of intermediate difficulty. If you are a skilled Proglog programmer it shouldn't take you more than 30-90 minutes to solve them. Problems marked with three asterisks (***) are more difficult. You may need more time (i.e. a few hours or more) to find a good solution.
I think the difficulty scale for the most part is comparable in Clojure.
P01 (*) Find the last element of a list.
Example:
user=> (last [1 1 2 3 5 8])
8
P02 (*) Find the last but one element of a list.
Example:
user=> (penultimate [1 1 2 3 5 8])
5
P03 (*) Find the Kth element of a list.
By convention, the first element in the list is element 0.Example:
user=> (nth2 2 [1 1 2 3 5 8])
2
P04 (*) Find the number of elements of a list.
Example:
user=> (length [1 1 2 3 5 8])
6
P05 (*) Reverse a list.
Example:
user=> (reverse [1 1 2 3 5 8])
(8 5 3 2 1 1)
P06 (*) Find out whether a list is a palindrome.
Example:
user=> (palindrome? [1 2 3 2 1])
true
P07 (**) Flatten a nested list structure.
Example:
user=> (flatten [[1 1] 2 [3 [5 8]]])
(1 1 2 3 5 8 )
P08 (**) Eliminate consecutive duplicates of list elements.
If a list contains repeated elements they should be replaced with a single copy of the element. The order of the elements should not be changed.Example:
user=> (compress "aaaabccaadeeee")
(\a \b \c \a \d \e)
P09 (**) Pack consecutive duplicates of list elements into sublists.
If a list contains repeated elements they should be placed in separate sublists.Example:
user=> (pack "aaaabccaadeeee")
((\a \a \a \a) (\b) (\c \c) (\a \a) (\d) (\e \e \e \e))
P10 (*) Run-length encoding of a list.
Use the result of problem P09 to implement the so-called run-length encoding data compression method. Consecutive duplicates of elements are encoded as tuples (N, E) where N is the number of duplicates of the element E.Example:
user=> (encode "aaaabccaadeeee")
((4 \a) (1 \b) (2 \c) (2 \a) (1 \d) (4 \e))
P11 (*) Modified run-length encoding.
Modify the result of problem P10 in such a way that if an element has no duplicates it is simply copied into the result list. Only elements with duplicates are transferred as (N, E) terms.Example:
user=> (encode-modified "aaaabccaadeeee")
((4 \a) (\b) (2 \c) (2 \a) (\d) (4 \e))
P12 (**) Decode a run-length encoded list.
Given a run-length code list generated as specified in problem P10, construct its uncompressed version.Example:
user=> (decode [[4 \a] [1 \b] [2 \c] [2 \a] [1 \d] [4 \e]])
(\a \a \a \a \b \c \c \a \a \d \e \e \e \e)
P13 (**) Run-length encoding of a list (direct solution).
Implement the so-called run-length encoding data compression method directly. I.e. don't use other methods you've written (like P09's pack); do all the work directly.Example:
user=> (encode-direct "aaaabccaadeeee")
((4 \a) (\b) (2 \c) (2 \a) (\d) (4 \e))
P14 (*) Duplicate the elements of a list.
Example:
user=> (duplicate "abccd")
(\a \a \b \b \c \c \c \c \d \d)
P15 (**) Duplicate the elements of a list a given number of times.
Example:
user=> (duplicate-n 3 "abccd")
(\a \a \a \b \b \b \c \c \c \c \c \c \d \d \d)

Sunday, November 08, 2009

Bleeding Edge Clojure Development using clj-gradle

Clojure is evolving very rapidly. I find myself using clojure and clojure-contrib from github most of the time. It's rather annoying to have to keep git pull and build them often. A few days ago, a CI server was setup for clojure.org to build both clojure and clojure-contrib and made them available in mvn style repo. Excellent, this means I can just use Gradle to handle it for me. For any new bleeding edge clojure project, all you need is the clj-gradle-1.0.0.jar and build.gradle file. If you don't have the clj-gradle-1.0.0.jar go here to see how to build it. Here is the updated version of the build.gradle file.
buildscript {
  repositories {
    flatDir(dirs:'lib')
  }
  dependencies {
    classpath name: 'clj-gradle-1.0.0'
  }
}

usePlugin(de.kotka.gradle.ClojurePlugin)
  repositories {
    flatDir(dirs:'lib')
  }

configurations {
  compileOnly {
    setVisible(false)
    setTransitive(false)
  }
  compile.extendsFrom(compileOnly)
}

repositories {
  mavenCentral()
  mavenRepo urls: "http://build.clojure.org/snapshots"
}

dependencies {
  compileOnly name: 'clj-gradle-1.0.0'
  compile 'org.clojure:clojure-lang:1.1.0-alpha-SNAPSHOT'
  compile 'org.clojure:clojure-contrib:1.0-SNAPSHOT'
}

task initProject(description: 'Initialize project directory structure.') << {
  ['clojure'].each {
  convention.sourceSets.all*."${it}".srcDirs*.each { it.mkdirs() }
  }
}

task copyToLib(dependsOn: configurations.default.buildArtifacts, type: Copy) {
  into('build/lib')
  from configurations.default
  from configurations.default.allArtifacts*.file
}
I'm using the shorthand syntax for dependencies. I also added mvn repo support. There is a separate entry to point to build.clojure.org. Two tasks were also added. One is initPrjoject. The initProject task was both inspired by Gradle cookbook as well as Hubert Klein Ikkink's blog post on mixed Groovy and Java using Gradle. The copyToLib was from the Gradle cookbook also. That's all there is to it. Gradle will now take care of all your build needs. If you want to use clojure 1.0 instead of the bleeding edge. Change the line "compile 'org.clojure:clojure-lang:1.1.0-alpha-SNAPSHOT'" to "compile 'org.clojure:clojure:1.0.0'". There isn't a mvn repo version of clojure-contrib though. At some point, the artifact id is changing from clojure-lang to just clojure as per this ticket. When that happens, just change your build.gradle file accordingly.

Friday, November 06, 2009

Wave Robot Using Groovy and Gaelyk Part 1 UPDATE

When I first saw Google Wave, all sorts of ideas came to me right away. I wanted to use Google App Engine and Google Wave to do some prototyping on some of the gaming ideas I have. Really, real time XMPP open system with the scalability of Google App Engine, what’s not to like? The question is, what tools to use. Right now Wave Robot has to be hosted on Google App Engine. I tend to use Groovy for rapid prototyping. That quickly narrows it down to Grails vs Gaelyk. I decided to go with Gaelyk because this really isn’t a traditional web application. Wave Robots interact with people through a series of servlets. Using Grails would be unnecessary for my needs. The approach Guillaume Laforge took with Gaelyk where he effectively groovifed the GAE API is a good fit. My plan is to split the code into three parts. Groovy servlets to handle Wave Robot interaction. Gaelyk to handle the GAE interaction such as DataStore, TaskQueue, Mail, and also to display some simple info if people come to the app itself. The final part is the Game AI part. Given GAE’s daily cpu quota, I don’t know if a complex AI engine will burn it up. I’ll find out soon enough. I’ll be using IntelliJ 8.1.3. It has terrific support for Groovy and Google App Engine. So let’s get started. You need to download the Google App Engine SDK if you don’t already have it. First let’s create a new Google App Engine project. For details on how to do that, check here Then we need to set it up for Gaelyk. Edit web/WEB-INF/appengine-web.xml You have to put in the application id you’ve registered for Google App Engine. Version number default is 1. That’s fine. Version number is a string, so you can put anything. you can put “theawesomeapp” if you like. If you do that, that particular version of the app will be available under the url theawesomeapp.latest.(yourappid).appspot.com. This let you test and deploy multiple versions of the app. Add the following to the file
<static-files>
<exclude path="/WEB-INF/**.groovy" />
<exclude path="**.gtpl" />
</static-files>
Now to edit web/WEB-INF/web.xml Add the following
<!-- The Gaelyk Groovlet servlet -->
<servlet>
<servlet-name>GroovletServlet</servlet-name>
<servlet-class>groovyx.gaelyk.GaelykServlet</servlet-class>
</servlet>

<!-- The Gaelyk template servlet -->
<servlet>
<servlet-name>TemplateServlet</servlet-name>
<servlet-class>groovyx.gaelyk.GaelykTemplateServlet</servlet-class>
</servlet>

<!-- Specify a mapping between *.groovy URLs and Groovlets -->
<servlet-mapping>
<servlet-name>GroovletServlet</servlet-name>
<url-pattern>*.groovy</url-pattern>
</servlet-mapping>

<!-- Specify a mapping between *.gtpl URLs and templates -->
<servlet-mapping>
<servlet-name>TemplateServlet</servlet-name>
<url-pattern>*.gtpl</url-pattern>
</servlet-mapping>

<!-- Define index.gtpl as a welcome file -->
<welcome-file-list>
<welcome-file>index.gtpl</welcome-file>
</welcome-file-list>
now create web/index.gtpl put the following in the file.
<html>
<body>
<p>
<% def message = "Gaelyk is working!" %>
</p>
<ul>
<% 3.times { %>
<li>${message}</li>
<% } %>
</ul>
</body>
</html>
under web/WEB-INF, create a groovy directory and a lib directory. edit web/WEB-INF/groovy/dostuff.groovy put the following in the file.
html.html {
body {
[1, 2, 3, 4].each { number -> p number }

p "Let's have a thumb war!"
}
}
you also need to put the following jars in web/WEB-INF/lib first the two jars from Google App Engine SDK:appengine-api-1.0-sdk-1.2.6.jar and appengine-api-labs-1.2.6.jar. the Groovy jar: groovy-all-1.6.5.jar Then the Gaelyk jar. I build gaelyk-0.3-SNAPSHOT.jar from github. Mostly because Guillaume hasn’t had time to build 0.3 yet. You can use 0.2 from Gaelyk site.. After this, add lib to your module library path. Congratulation, your Gaelyk environment is all set. Run it, and you’ll see the output “Gaelyk is working!” 3 times. That comes from index.gtpl. Now go to http://localhost:8080/dostuff.groovy you’ll see the output from the dostuff.groovy Groovlet. With Gaelyk setup out of the way, it’s time to get to the Wave Bot. First you need to download a few files from Wave Robot Java Client Library. Get all 4 of them and place them under WEB-INF/lib. We only need 3 of them right now. But oauth will come in handy later. Create a new directory called _wave under web. And create a file web/_wave/capabilities.xml,
<?xml version="1.0"?>
<w:robot xmlns:w="http://wave.google.com/extensions/robots/1.0">
<w:capabilities>
<w:capability name="blip_submitted"/>
</w:capabilities>
<w:version>Alpha1</w:version>
</w:robot>
This is where we tell Wave which events our Wave Robot cares about. In this case, we want blip_submitted. This event occurs everytime someone “completes” a blip by either clicking on “Done”, or hit shift-enter. There is also a version number for the Wave Robot. This version number has NOTHING to do with the Google App Engine version number. Everytime you create a new version of the Wave Robot you have to change this. Because Google Wave caches your robot. Again, it’s a string. So you can put whatever you want. Now to create the servelets that let our Wave Robot to pretend to be human. Under src, create a groovy directory. First create src/groovy/ProfileServlet.groovy using the following
import com.google.wave.api.ProfileServlet

class ProfileServlet extends com.google.wave.api.ProfileServlet {

  String RobotName="Wave Nakama"
  String RobotAvatarUrl="http://wavenakama.appspot.com/images/wnlogo.gif"
  String RobotProfilePageUrl="http://wavenakama.appspot.com/"

}
This servlet is called whenever someone try to get the profile on your Wave Robot. Then create src/groovy/SayanythingServlet.groovy using the following
import com.google.wave.api.*

class SayanythingServlet extends AbstractRobotServlet {

    void processEvents(RobotMessageBundle bundle) {
        def wavelet = bundle.wavelet
        if (bundle.wasSelfAdded()) {
            def blip = wavelet.appendBlip()
            def textView = blip.document
            textView.append("Your wish is my command!")
        }
        for (event in bundle.events) {
            if (event.type == EventType.BLIP_SUBMITTED) {
                def blip = wavelet.appendBlip()
                def textView = blip.document
                textView.append("Tell me more about ${event.blip.document.text} Please. You are so interesting!")
            }
        }
    }
The servlet responds to two events. The first was is “wasSelfAdded”. This happens when the Wave Robot is first added to the Wave. Think of it as your Wave Robot’s greeting. The second one is what we marked back in the capabilities.xml, BLIP_SUBMITTED. You can access the blip that just occurred in event.blip.document.text. Now we have to edit the the web.xml again to register the two servlets. Google Wave expects the Wave Robot to repsond to specific urls. Add the following to the web.xml
<servlet>
<servlet-name>Sayanything</servlet-name>
<servlet-class>SayanythingServlet</servlet-class>
</servlet>

<servlet-mapping>
<servlet-name>Sayanything</servlet-name>
<url-pattern>/_wave/robot/jsonrpc</url-pattern>
</servlet-mapping>

<servlet>
<servlet-name>Profile</servlet-name>
<servlet-class>ProfileServlet</servlet-class>
</servlet>

<servlet-mapping>
<servlet-name>Profile</servlet-name>
<url-pattern>/_wave/robot/profile</url-pattern>
</servlet-mapping>
The Profile servlet must respond to the url /_wave/robot/profile. The Wave Robot communicate with the Wave system using /_wave/robot/jsonrpc That’s it. Right now there is no way to test Wave Robot locally. However, you can see if the Wave Robot is responding to the Profile call correctly by going to http://localhost:8080/_wave/robot/profile. If everything is working, you’ll see a JSON output of the Wave Robot profile. I created an image to be used as the Wave Robot profile picture and placed it under web/images. Final step, time to upload the application to Google App Engine. When it’s done, try it out by going to (applicationid).appspot.com to verify your app is working. Then create a new Wave. Add your Wave Robot as a participant by using the address (applicationid)@appspot.com. For example, in my case it’s wavenakama@appspot.com. If all goes well, you now have the basic Gaelyk and Wave Robot setup for you to experiment with. One thing you’ll notice is the Profile url for the Wave Robot doesn’t appear to be working. I’m guessing it’s a bug of the API. UPDATE: Thanks to Tim Yate who tracked this done. The official code is wrong. It's RobotProfilePageUrl, not RobotProfileUrl. I've corrected the code sample, and filed an issue with Google.

Monday, November 02, 2009

Gradle + Clojure = clj-gradle

I'm not a maven fan. I use it with lift, but that's pretty much it. In short, I only use it when I'm forced to. So when Gradle came along, I adapted it pretty early on. If you don't know what Gradle is. Check it out here. It has extensive documentation, which is very impressive considering the project is so young. When Meikel Brandmeyer announced he wrote clj-gradle, I really wanted to try it out. I finally found time to give it a spin. I ran into a few bumps, but I did get it to work. Here is what I did, step by step. Get the source. You need mercurial. Use "hg clone http://bitbucket.org/kotarak/clj-gradle/" In the directory, "mkdir lib", then put clojure-1.0.0.jar in lib. Check the build.gradle file, the version number should be set to 1.0.0 Issue "gradle build". You should get a build successful. The file clj-gradle-1.0.0.jar is created under build/libs. You are done. Time to try it out. Create a new sandbox directory. Then issue "mkdir lib", you need to put both clj-gradle-1.0.0.jar clojure-1.0.0.jar in lib. I followed the README and created src/main/clojure/test/example.clj, then pasted in the code from it.
(ns test.example
  (:gen-class))

(defn -main
  []
  (println "Hello, World!"))
I then use the build script from README, with a few changes to put in specific version number.
buildscript {
  repositories {
    flatDir name: 'lib', dirs: 'lib'
  }
  dependencies {
    classpath name: 'clj-gradle-1.0.0'
  }
}

usePlugin(de.kotka.gradle.ClojurePlugin)
  repositories {
    flatDir name: 'lib', dirs: 'lib'
  }

configurations {
  compileOnly {
    setVisible(false)
    setTransitive(false)
  }
  compile.extendsFrom(compileOnly)
}

dependencies {
  compileOnly name: 'clj-gradle-1.0.0'
  compile name: 'clojure-1.0.0'
}
Now issue "gradle build", you should get a build successful. To run it, I used "java -cp lib/clojure-1.0.0.jar:build/libs/cljgradletest-unspecified.jar test.example" I used cljgradletest-unspecified.jar because the name of my sandbox directory is cljgradetest. All goes well, you'll see the "Hello, World!" output. Congratulation, your clojure project is now backed by Gradle.