Archive for December, 2006

Ferret’s what we use as our search engine. It’s the Ruby port of Lucene. We really like it & would recommend it to others.

But a while ago we ran into a problem with Ferret. The index Ferret created would sometimes be corrupted upon system activation. It may have been the version, it may have been the manner of system generation.

We fixed the problem by removing the pre-built index. Instead, the fist search conducted on a new system would trigger a “rebuild” for all content in the system.

This seemed like a good solution at the time, as it guaranteed a reliable demo experience. There was just 1 tiny problem – the “rebuild” took time. And that meant the first user experience with search was by far the slowest.

Well, time passed and we added more content. Which in turn meant the rebuild took longer. Depending on overall system utilization it can take anywhere from 25s to 45s. And that’s a long time for your 1st search!

So, due to popular request, the pre-built Ferret index is back & appears to be copying reliable. 1st search performance has gone from 30s to around 1s.

And to be frank, we should have made this change a lot sooner.. Enjoy the speedier demo experience!

-Dave

p.s. It’s likely the same strategy will be applied to our next Open Source Edition distro.

Some say the no-click requisition is the coolest feature in Coupa eProcurement. We’re pretty proud of it. Here’s a little intro to the feature and a demo.

This 5-minute video will show you all about how policies guide users in Coupa eProcurement and how to create policies easily.

We have been using the Dojo toolkit right from the beginning, but back then we only used the Editor widget, and only on one page. From there, we gradually added Dojo functionality, with Dialogs on a few more pages, and just recently, Buttons (custom ones at that) on every page! The toolkit has become an integral part of our application, and we realized the hard way that the number of objects downloaded gets really big really quickly if you don’t have a good profile.

There was documentation on profiling Dojo, and documentation on custom widgets, but numerous sources of information and too much code reading was necessary to figure out how to profile Dojo with a custom widget! Hopefully, writing it all down here will benefit those who come after us…

Profiling

Dojo is a very big toolkit. Chances are, your app won’t use half the features, so the toolkit has a small core, dojo.js, that knows how to pull in components as necessary. This makes the initial install trivial. The problem is, as you use more of the toolkit, you are making the user pull down many small files, and this greatly increases latency, not to mention browser caching is defeated in many cases. Dojo’s solution is a concept of profiling, where you define the objects that you need, and bake them into a custom compressed dojo.js file. This way the browser gets all/most of the code in a single, compressed, and cacheable download.

Sources

The first step to creating our own Dojo package, is to download the source:

svn co http://svn.dojotoolkit.org/dojo/tags/release-0.4.1

Conventions

Next, we will write our very own Coupa Button. Dojo gives you some flexibility in defining custom widgets, but we’re Rails developers, so convention triumphs over configuration, and the dojo conventions for custom widgets are:

1. the widget should be called <you>.widget.<Component>, so in our case, coupa.widget.Button
2. the code should be placed in release-0.4.1/../<you>/widget/<Component>.js
3. the HTML/CSS associated with the widget should go in release-0.4.1/../<you>/widget/templates
4. the images associated with the widget should go in release-0.4.1/../<you>/widget/templates/images

We end up with with this directory structure:

Code

The conventions continue into the code for the custom widget. We only wish to change the CSS and the images used by Dojo’s Button class, and by following conventions, we minimize the code we have to write in our Button.js:

dojo.provide("coupa.widget.Button");
dojo.require("dojo.widget.Button");

dojo.widget.defineWidget(
    "coupa.widget.Button",
    dojo.widget.Button,
    {
        templateCssPath: dojo.uri.moduleUri("coupa", "widget/templates/ButtonTemplate.css"),
        inactiveImg: "../coupa/widget/templates/images/coupaButton-"
    }
);

It almost reads like english: we are providing a coupa.widget.Button, and to do that, we need dojo.widget.Button. We define the former as a subclass of the latter, overriding two of the attributes. Short and sweet. Also note the use of the dojo.uri.moduleUri helper made possible by the conventional directory structure.

Profile

We then create a profile release-0.4.1/buildscripts/profiles/coupa.profile.js:

var dependencies = [
    "dojo.widget.Dialog",
    "dojo.widget.DropdownDatePicker",
    "dojo.widget.Button",
    "coupa.widget.Button"
];

dependencies.prefixes = [
    ["coupa", "../coupa"]
];

load("getDependencyList.js");

Again, the minimum information needed to do the job: we want to include these objects, and find them in this path prefix in addition to Dojo’s.

Now, you might have noticed a file called manifest.js in our coupa directory in the screenshot above. This file tells Dojo’s dynamic class loader what components live under a directory, so it knows to look here when it needs these components. We don’t need this file now that our button is baked into dojo.js, so we won’t elaborate on it in this article, but it is good for testing our widgets, in which case we use the minimal dojo.js.

Finally we are ready to generate our very own dojo.js:

cd release-0.4.1/buildscripts
ant -Dversion=0.4.1 -Dprofile=coupa -Ddocless=true release intern-strings

This will read the object list defined in profiles/coupa.profile.js and prepare a release without documentation, including associated HTML/CSS objects.

Deploy

Now we can deploy our new Dojo package:

mkdir $RAILS_ROOT/public/javascripts/dojo
cd ../release/dojo
cp -r dojo.js iframe_history.html src nls $RAILS_ROOT/public/javascripts/dojo

and let’s not forget our custom widget:

cd ../../..
cp -r coupa $RAILS_ROOT/public/javascripts

Why deploy the huge Dojo source tree and our full code structure when all the interesting stuff is already in dojo.js, you ask? By convention, Dojo is deployed with its full source tree, and so should custom extensions. Our images need to be deployed in that location, for one, and our code needs to be in source control somewhere anyway. Deploying them in the standard locations also allows us to debug them by simply substituting in the minimal dojo.js. Lastly, we might want to use a widget, like Editor2, in only a few places, and don’t wish to enlarge our dojo.js with it, in which case we can just leave it out of the profile, and it will be loaded from the source tree.

Instantiate

We’ve come to the very last step, which is, of course, to use our new button:

<div id="do_stuff_button" dojo:type="coupa:Button" onclick="doStuff()">Hit me, Baby!</div>

and for buttons that are inserted into a page through AJAX we need to manually instantiate them:

dojo.widget.createWidget('do_stuff_button')

Conclusion

That’s all for today, folks!

Dojo is not for everyone. In fact, there’s a different library out there for each developer’s needs, it seems. But those whose needs are met by the Dojo toolkit, will no doubt appreciate the efforts of Alex Russell and his team in creating an astoundingly comprehensive set of tools and widgets with incredible consistency and quality, which is all the more amazing given how much time they spend babysitting n00bs like myself on #dojo!

PS click here to get straight to the answer

There doesn’t seem to be any documentation on getting incoming emails into a Rails app running on Windows. I’ve tried every combination of search keywords I can think of, having just learnt a lesson on how they can determine your success in finding an elegant solution that someone put up recently. So if you know of a good guide somewhere else, please kindly post a comment.

The closest I found was this entry in the Ruby on Rails wiki. Virtually the entire page is about working with UNIXy MTAs and, to my surprise and horror, none of them has been satisfactorily ported over to Windows! There is a little section on “Receiving with ActionMailer on windows” but it involved manually saving the email in a file and sending the file to the Mailer as input, not the best thing to have our customers do!

Long story short, I eventually found two decent-looking free SMTP servers for Windows:

1. XMail – Open Source
2. MailEnable Standard Edition – Freeware

Champions of Open Source that we are, XMail just had to get first dibs. It turned out MailEnable’s scripting didn’t look very easy anyway.

So here’s how you receive email in a Rails app running in Windows:

First of all, install XMail using this forum post. The XMail Wizard step is only for setting the admin username/password, so if you cannot open the rar file that it comes in, just use xmcrypt to encrypt your admin password:

c:\mailroot\bin\xmcrypt <password>

and write this in the ctrlaccounts.tab file:

"<username>"    "<xmcrypted password>"

BE SURE to use quotes around each token, a tab between the two, and a newline at the end of the line!!

Create a domain for your app in XMail Administrator, as per the instructions, and in this domain, add the accounts that correspond to your mailers, then for each of these accounts, uncheck the Store mail locally checkbox. Your screen should now look like this:

Now for each of these accounts, click into the Advanced tab and enter the following into the mailproc.tab settings:

"external","0","60","ruby","<RAILS_ROOT>/script/runner","-e","production","<YourMailer>.receive(File.read(ARGV[2][4..-1]).sub(/\A.*?<<MAIL-DATA>>/im, '').lstrip)","$(FILE)"

substituting your own RAILS_ROOT and the mailer class that corresponds to the account. As you can see, there a little bit of massaging needed on both the file name and the email content that XMail provides.

Set the MX record for the domain to point to the XMail host and your hungry mailers will be fed!

Big thanks to Davide Libenzi for creating the amazing XMail, which combines SMTP, POP3, IMAP, Finger, an admin server for windows in one tight package, as well as to BooT on the XMail forum for the instructions that probably saved me days of suffering.

A few days ago I checked in an initial cut at multicurrency support for Coupa eProcurement. It’s based somewhat on Tobias Luetke’s Money gem, but with currency & exchange rate info pulled out into tables and amounts stored as floats for subcent pricing, among other changes. It’s available in SVN, of course, but it still needs more polishing and testing before it’ll make it into a release.

One of the main challenges in adding this support, and in extending the application in general, is to keep complexity optional and necessary maintenance to a bare minimum. Exchange rates are a good example of why this can be tricky, since they always need to be reasonably up-to-date, but companies get them from all sorts of different places and in different formats. The application integration side of the problem has even spawned aggregation & normalization businesses like Cloanto’s Currency System. While we shouldn’t rely on a particular commercial service for bringing in data, we also don’t want to leave the integration as an exercise for the implementor, as is done with many ERP implementations, so this is an area where I’d like to do more work. So far we have a CSV file loader that can import rates in the same way as all of our other loaders, but that’s not very interesting. I’ve also implemented (but not exposed) a SOAP callout to the Federal Reserve Bank of New York to automatically populate a number of rates, and that’s far more promising. Going forward I’d like to support a number of different web service-based rate sources, starting with the free/governmental ones, and let the admins configure what gets pulled in when, with the goal of enabling one-time setup and free data for all but the most demanding customers, with easy integration of commercial services for the rest. Ideas? Send ‘em to david at coupa.com.

We have been receiving multiple customer requests for integrating Coupa with Microsoft Active Directory, primarily for Single-Sign On purposes, so we decided it was worth investigating.

Considering Active Directory implements LDAP, which is quite a mature standard, we thought it’d be a piece of cake to implement. In the end it turned out to be very easy, but the whole LDAP association clouded my search abilities, and I ended up spending some time looking at some excellent but pretty low level libraries and tips. Once I used the terms “activedirectory” and “active directory” in my searches, the rest was trivial…

This is all thanks to Justin Mecham’s ActiveDirectory gem. Not only does it wrap ActiveDirectory objects in an ActiveRecord-like interface, more importantly for us, it provides a built-in method for authenticating a user against the directory! So all we had to do was:

ActiveDirectory::User.find(username).authenticate(password)

Even when we thought it’d be a piece of cake, we didn’t think it’d be a one-line piece of cake! Granted there are a bunch of exceptions you have to catch and stuff, but it’s a pretty trivial solution to what could be seen as a pretty advanced feature. I am also going to document the setup required for this piece of magic to work.

First of all, you need to install the underlying Ruby/LDAP package. If you’re running on Windows, you get to install Jan Szumiec’s gem:

gem i http://jps.kni.pk.edu.pl/files/ldap-0.9.7-mswin32.gem --include-dependencies

There’s also a binary distribution in the form of a RPM, but everybody else will have to compile it from source.

Next, we install the activedirectory gem itself:

gem i activedirectory --include-dependencies

And we add the Active Directory settings to the environment.rb:

require_gem 'activedirectory'
ActiveDirectory::Base.server_settings = {
  :host     => "directory.example.com",
  :port     => 389,
  :username => "username",
  :password => "password",
  :domain   => "example.com",
  :base_dn  => "DC=example,DC=com"
}

Those of you are currently using the Login Engine by James Adam can also refer to our extension that authenticates against Active Directory when it is available. If you can’t wait for the Active Directory code to get merged back into trunk, grab the active directory branch:

svn co http://svn.coupa.com/coupa_engine/branches/activedirectory-sso

That’s it, folks!

UPDATE – Running as a service

Just found out that the ActiveDirectory gem creates its own Logger and so when you try to run your new ActiveDirectory-enabled Rails app as a Win32 service, it will fail when the ActiveDirectory module first tries to connect to a directory server! Much hair was lost to bring you this piece of information…..

Do the following in your environment.rb together with the server settings:

ActiveDirectory::Base.logger = RAILS_DEFAULT_LOGGER

and you will be fine. Of course, you could use a different logger if you wish.

Because we develop extensively on the Mac, it is quite ironic that our Mac distribution is the least straightforward to install and use. In fact, the Windows distribution, with one click to install, and two clicks to run, is so easy that we couldn’t bear to wrap it in a Wizard-style installer.

So we are always on the lookout for better ways to package up our Mac distribution, it’s just hard to commit to spending too much time on packaging when there’s so much product to build.

Currently, we bundle our Mac distribution with a customized version of Ryan Raaum’s wonderful Locomotive package, which allowed us to bundle all the Rails prerequisites in a single package. But the user still needs to perform numerous steps before they finally see their Coupa instance in their browser.

Duane Johnson found the situation unsatisfactory too, but I actually thought the links that he found, as well as the one put up in a comment, are all pretty good. Maybe our 4th release might contain a Mac distribution that we’d be proud of, look out for it…

Getting the latest source code for Coupa Open Source isn’t as hard as you might think. Of course, if you want our previously published releases, you can get them from Sourceforge. But for those that want to see the stuff as it is checked-in, I figured I would explain from a “non-technical” perspective. If you aren’t a novice like myself and already have things like mySQL installed, then you can just check out the Trac Wiki site. These are instructions for someone on Windows. If you’d like Linux or Mac instructions, leave a comment and I’ll do those too. Before you try to download the code, here are a few things you need to do:

1. Install TortoiseSVN – It’s a very easy Windows shell extension for accessing the source code. Download here
2. Install MySQL – You can choose other databases, but this one is straightforward. I’d do the Community Server (open source) for currently the latest stable edition 5.0 and pick the Windows Essentials install here. I don’t have all the install steps, but follow the defaults. Recommend that you choose the defaults or set them to have username of root, hostname of localhost (127.0.0.1), port 3306, and no password.
3. Create a new schema in mySQL – After install of MySQL, use the MySQL Administrator to log into MySQL and create a schema for Coupa. To do this, once logged in, you’ll see a bunch of options on your left hand side of MySQL administrator, click catalogs. A “Schemata” list will drop-down. Right click in the list of schemas (which should have a couple like mysql and test) and pick “Create new schema”. Enter “coupa_development” for the name of the new schema.
4. Install Ruby. You can get our programming language by going here. I’d pick the Windows installer which will take you to the download page on RubyForge. I did the 1.8.4-19 Final release (the .exe one). Install that program. You’ll be asked what additional components to install. Make sure the Enable RubyGems is installed. The others are optional. After installing, I had to set the PATH to include the key newly created Ruby directories…but others said that this was taken care of automatically. If you need to set them, use the Environment Variables (in Systems Properties) to include your location of install. For me, that was C:\ruby;C:\ruby\bin
5. Install additional engines for Ruby. Open up a cmd.exe window. Type in the following:
   • “gem install ferret”. BTW, you’ll need to pick the latest ferret that is supported and appropriate for your environment, which currently is 0.10.9 (mswin32)
   • “gem install vpim”
   • “gem install rails –include-dependencies”

You’re about 75% of the way there! Now, let’s fetch the code. First, create a folder on your local drive…in this case, we’ll create C:\CoupaOS. Now, navigate to that folder and right click on it. You should see “TortoiseSVN” in the list. Pick TortoiseSVN which will open a subfolder (or sub-option). On the top of that, you should be able to pick “Checkout…”. Type in “http://svn.coupa.com/coupa/trunk” as the URL of the repository, which is where our source code is. Select “OK” and it should start fetching all the source code. After a couple minutes, you’ll have the whole application on your machine.

The next step is to load the schema in MySQL on your machine. Here are the steps:

1. Open a cmd.exe window and navigate to the CoupaOS directory. Type “copy config\databases\database.mysql config\database.yml”. This tells the system that MySQL is the database of choice.
2. Type “rake db:schema:load”
3. Type “rake db:seed:engines:load”

Now, you are ready to run. From now on, everytime you want to startup Coupa eProcurement on your machine, all you need to do is open a cmd.exe window, navigate to C:\CoupaOS and type “ruby script/server”. That turns everything on. Now go to your web browser of choice and type in “http://localhost:3000/”. You’ll be asked to log in. “williams” is the username of the system administrator with “welcome” as the password. Three things of note:

1. Let’s say, a week later, you want to grab all the new checkins. You use Windows and navigate to the CoupaOS folder and right click. You should see “SVN Update…” as an option. Pick that and the system should download all deltas. Occasionally, we may have some schema changes which require some additional steps. You are taking the latest and that has risks
2. If you want to use the bulk loader, which allows you to load in things like users, catalog items, accounts, suppliers, etc., you need to start the bulkloader. Before you type in “ruby script/server”, you’ll need to type in “rake backgroundrb:start” to get the loaders up and going.
3. We rely on an smtp server to send notifications (e.g., approval notifications) out of Coupa to your email system. If you don’t set it up, notifications will still end up in the notification inbox of the Coupa user (they just won’t get sent to your email inbox).

I’ll also be posting this in the Coupa Forums. We’d love to see you posting questions/comments/thoughts/whatever in our Forums!

We’re wrestling with how to tie Coupa eProcurement into Active Directory for Single Sign-On support for Windows environments.

It all comes down to whether to use IIS and it’s built in authentication layer to provide a “no code” LDAP & Single Sign-On capability. The downside is that we’d be using the Ruby binding to IIS which isn’t as mainstream as we’d like. A second downside is that instead of a nifty “pack of Mongrels” to execute the Ruby code we’d be using FCGI in IIS.

The other route is to stick whit Apache 2.2 with a pack of Mongrels and poke into Active Directory via LDAP calls. But that integration option requires knowing the “topology”, or how Active Directory’s LDAP is structured on a per customer basis.

Here are the details of what we are currently considering for customers who already have a Microsoft network in place with IIS and Active Directory:

1. FCGI behind IIS

   Pros: Quite common, keep existing IIS, free Single Sign On

   Cons: IIS headaches, FCGI headaches, discouraged in Rails community

2. Mongrel behind RForward in IIS

   Pros: Keep existing IIS without FCGI headaches, free Single Sign On

   Cons: RForward not maintained anymore

3. Mongrel behind Apache 2.2

   Pros: Common, we have experience

   Cons: Require customer to change infrastructure, need to integrate with Active Directory

RForward looks like a very attractive option, but we will need to try it out to see how stable it is. Since it is open source, and was itself developed in an open source tool, we might even be able to fix issues we find!