Automate Your Database Backups

I have a few blogs and other applications that leverage MySQL as a database. I wanted to automate the backup of the associated database data. My specific requirements are:

  • Needs to work on Windows Server (my hosting machine).
  • My database is MySQL.
  • It needed to be automatic. In other words, no human interaction.
  • I wanted it to happen twice per day. This minimized the risk of loss to 12 hours
  • The backup couldn’t reside on the same machine. It had to be more durable.
  • I wanted an alert that the backup occurred.

After some searching I found the following PHP script that would export the database to a file.

$fileName = 'dbName'.date("Y-m-d").'.sql';
$folder = 'c:\\temp\\';
$fullName = $folder.$fileName;

$cmd = 'mysqldump --user='.DB_USER.' --password='.DB_PASS.' --host='.DB_HOST.' --port='.DB_PORT.' dbName > '.$fullName;

system($cmd);

This bit of script executes the MySql command to backup a database. This could probably be done other ways (e.g. batch file), but this was convenient for testing. I could manually navigate to the PHP file with this script in a browser.

This got exported the SQL data to a file on the local disk (temp folder in the above case). Now I needed to get it off of this system. Each of the database files are fairly small, so I chose to simply email them to my gmail account. If I got into a situation where these were bigger or I would probably offload them either to my DropBox or AWS S3 storage. For now they are small and the email serves as an alert that the backup occurred. Here is a bit of code that emailed the backup file:

$mail = new PHPMailer(true);

$mail->IsSMTP();
$mail->Host = "mail.diglabs.com";
$mail->SMTPAuth = true;
$mail->SMTPSecure = "ssl";
$mail->Host = "smtp.gmail.com";
$mail->Port = 465;
$mail->Username = "username@diglabs.com";
$mail->Password = "password-for-username";
$mail->AddReplyTo('username@diglabs.com');
$mail->SetFrom('username@diglabs.com');
$mail->Subject = 'Database Backup';
$mail->AltBody = 'To view the message, please us an HTML compatible email viewer!';
$mail->MsgHTML('Database backup is attached.');
$mail->AddAttachment($fullName, $fileName);
$mail->Send();

// Delete the unzipped file from your server
//	Note: unlink command did not work here.
system("del $fullName");

echo 'Backup completed, email sent';

In the above code I am using PhpM@iler as an email transport class. In this particular example, I am sending an email to ‘username@diglabs.com’ with the database back up file attached. The domain ‘diglabs.com’ is my domain and I use Google Apps for email. The above code sends email via Google Apps (smtp.gmail.com, port 465, ssl). Finally, I remove the backup file from the local disk.

Hope you find this useful. If you have suggestions or improvements, please leave them in the comments.

Posted in php | 1 Comment

Python Script to Build, Test and Pack .NET Projects

At work we use Jenkins as our continuous integration server. Our implementation at work has to integrate with ClearCase and does a few tasks before actually doing the build. This is pretty typical. Our setup then calls an external script to build. We are not using some of the features that can be configured for Hudson. This may not be as stream-lined as it can be but it has some benefits and I have to work within these constraints.

Since our build machine already has Python installed, I thought I would see if I can put together a reusable build script. I played around a bit and landed upon the following to call build (MSBuild), execute tests (MSTest) and create Nuget packages. I thought I would share this with you.

# Generic build script that builds, tests, and creates nuget packages.
#
# INSTRUCTIONS:
#    Update the following project paths:
#        proj        Path to the project file (.csproj)
#        test        Path to the test project (.csproj)
#        nuspec        Path to the package definition for NuGet.
#
#        delete any of the lines if not applicable
#
#
#    Update the paths to the build tools:
#        msbuild        Path to msbuild
#        test        Path to mstest.exe (requires Visual Studio) (optional – set to None)
#        nuget        Path to nuget.exe (requires NuGet command line tool) (optional - set to None)
#        trx2html    Path to trx2html.exe (http://trx2html.codeplex.com/) (optional - set to None)
#
#
# USAGE:
#
#     proj = r'path to project (.csproj)'
#     test = r'path to project containing test (.csproj)'
#     nuspec = r'path to nuspec definition (.nuspec)'
#
#
#     msbuild = r'C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe'
#     mstest = r'C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe'
#     nuget = r'C:\BuildTools\nuget\2199eada12ce\nuget.exe'
#     trx2html = r'C:\BuildTools\trx2html\0.6\trx2html.exe'
#
#     bld = MsBuilder(msbuild, mstest, nuget, trx2html)
#     bld.run(proj, test, nuspec)
#

import os, shlex, subprocess, re, datetime    

class MsBuilder:
    def __init__(self, msbuild=None, mstest=None, nuget=None, trx2html=None):
        # The following dictionary holds the location of the various
        #    msbuild.exe paths for the .net framework versions
        if msbuild==None:
            self.msbuild = r'C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe'
        else:
            self.msbuild = msbuild

        # Path to mstest (this requires vs2010 to be installed
        if mstest==None:
            self.mstest = r'C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe'
        else:
            self.mstest = mstest

        # Path to nuget packager
        if nuget==None:
            self.nuget = r'C:\BuildTools\nuget\2199eada12ce\nuget.exe'
        else:
            self.nuget = nuget

        # Path to trx2html transformation tool
        if trx2html==None:
            self.trx2html = r'C:\BuildTools\trx2html\0.6\trx2html.exe'
        else:
            self.trx2html = trx2html

    def build(self, projPath):
        # Ensure msbuild exists
        if not os.path.isfile(self.msbuild):
            raise Exception('MsBuild.exe not found. path=' + self.msbuild)

        arg1 = '/t:Rebuild'
        arg2 = '/p:Configuration=Release'
        p = subprocess.call([self.msbuild, projPath, arg1, arg2])
        if p==1: return False    # exit early

        return True

    def test(self, testProject):
        if not os.path.isfile(self.msbuild):
            raise Exception('MsBuild.exe not found. path=' + self.msbuild)
        if not os.path.isfile(self.mstest):
            raise Exception('MsTest.exe not found. path=' + self.mstest)

        # build the test project
        arg1 = '/t:Rebuild'
        arg2 = '/p:Configuration=Release'
        p = subprocess.call([self.msbuild, testProject, arg1, arg2])

        # find the test dll
        f = open(testProject)
        xml = f.read()
        f.close()
        match = re.search(r'<AssemblyName>(.*)</AssemblyName>', xml)
        if not match:
            print 'Could not find "AssemblyName" in test project file.'
            return False

        outputFolder = os.path.dirname(testProject) + '\\bin\\Release\\'
        dll =  outputFolder + match.groups()[0] + '.dll'
        resultFile = outputFolder + 'testResults.trx'
        if os.path.isfile(resultFile):
            os.remove(resultFile)

        # execute the tests in the container
        arg1 = '/testcontainer:' + dll
        arg2 = '/resultsfile:' + resultFile
        p = subprocess.call([self.mstest, arg1, arg2])

        # convert the results file
        if os.path.isfile(self.trx2html):
            subprocess.call([self.trx2html, resultFile])
        else:
            print 'TRX to HTML converter not found. path=' + self.trx2html

        if p==1: return False # exit early

        return True

    def pack(self, packageSpec, version = '0.0.0.0'):
        if not os.path.isfile(self.nuget):
            raise Exception('Nuget.exe not found. path=' + self.nuget)

        outputFolder = os.path.dirname(packageSpec) + '\\artifacts\\'
        if not os.path.exists(outputFolder):
            os.makedirs(outputFolder)

        p = subprocess.call([self.nuget, 'pack', packageSpec, '-Version', version, '-Symbols', '-o', outputFolder])

        if p==1: return False #exit early

        return True

    def validate(self, projectPath):
        packFile = os.path.dirname(projectPath) + '\\packages.config'
        if os.path.isfile(packFile):
            f = open(packFile)
            xml = f.read()
            f.close()
            print xml
            match = re.search(r'version="0.0.0.0"', xml)
            if match:
                # Found a non-versioned package being used by this project
                return False
        else:
            print 'No "packages.config" file was found. path=' + packFile

        return True

    def run(self, proj=None, test=None, nuspec=None):
        summary = '';

        # File header
        start = datetime.datetime.now()
        print '\n'*5
        summary += self.log('STARTED BUILD - ' + start.strftime("%Y-%m-%d %H:%M:%S"))

        # Build the project
        if proj is not None:
            buildOk = self.build(proj)
            if not buildOk:
                self.log('BUILD: FAILED', start)
                sys.exit(100)
            summary += self.log('BUILD: SUCCEEDED', start)
        else:
            summary += self.log('BUILD: NOT SPECIFIED')

        # Build the tests and run them
        if test is not None:
            testOk = self.test(test)
            if not testOk:
                print self.log('TESTS: FAILED', start)
                sys.exit(100)
            summary += self.log('TESTS: PASSED', start)
        else:
            summary += self.log('TESTS: NOT SPECIFIED')

        # Package up the artifacts
        if nuspec is not None:
            packOk = self.pack(nuspec, '0.0.0.0')
            if not packOk:
                print self.log('NUGET PACK: FAILED', start)
                sys.exit(100)
            summary += self.log('NUGET PACK: SUCCEEDED', start)
        else:
            summary += self.log('NUGET PACK: NOT SPECIFIED')

        # Validate dependencies
        if not self.validate(proj):
            print self.log('DEPENDENCIES: NOT VALIDATED - DETECTED UNVERSIONED DEPENDENCY', start)
            sys.exit(100)
        summary += self.log('DEPENDENCIES: VALIDATED', start)

        # Build footer
        stop = datetime.datetime.now()
        diff = stop - start
        summary += self.log('FINISHED BUILD', start)

        # Build summary
        print '\n\n' + '-'*80
        print summary
        print '-'*80

    def log(self, message, start=None):
        timestamp = ''
        numsecs = ''
        if start is not None:
            split = datetime.datetime.now()
            diff = split - start
            timestamp = split.strftime("%Y-%m-%d %H:%M:%S") + '\t'
            numsecs = ' (' + str(diff.seconds) + ' seconds)'
        msg = timestamp + message + numsecs + '\n\n'
        print '='*10 + '> ' + msg
        return msg

You can also find this script on github. I am not a Python guru, so if you see opportunities for improvement provide comments on the github page.

Posted in build, ci, python | Leave a comment

Busy, Busy, Busy…

Posted in agile | 3 Comments

4 Principles To Estimation Applied To Software Development

Estimating when a new software feature will be done is a tricky but important task. “When will you be done?” Typically on the other end of this question is the person writing your check. Maybe not directly, but follow the request up the chain and you will certainly end up eye to eye with the check writer. Another important business fundamental is predictability of the development pipeline. Voice of the customer goes in one end, software features come out the other end. The ability to estimate how long the pipeline will be tied up on a feature is important for scheduling resources and planning. Thankfully, applying some fundamentals will provide faster more accurate measurements.

Fundamentals of Estimation

First let’s look at a few examples from every day life. Here is the first estimation problem:

An object traveling exactly 100 miles per hour is traveling a distance of exactly 100 miles.

Estimate how long it will take for the object to travel the distance. A bit of simple math (remember high school physics?)

image Solving for time we get

image In this case we plug-in 100 miles for the distance and 100 miles per hour for the velocity and we calculate 1.0 hours as the time for the object to travel the distance. What is the certainty we have in our estimate? In the above problem, both the distance and the velocity were exactly known so the accuracy of our estimate is 100%. The error is zero.

This is really not an estimate, but more of a calculation. Estimate generally involve some approximations. However, from this simple problem we can already see some fundamentals of estimation that we will use. The relation ship between time, velocity and distance will be useful.

In the case of software development ‘distance’ is usually a measure of total size (or effort) of the features to be estimated. A common measure of size is ‘story points’. Mike Cohn has a great blog post on ‘story points’. Software development teams can also measure their ‘velocity’ in terms of ‘story points per unit of time’. For example in scrum the velocity might be measure as 40 story points per sprint. Sprint are always time-boxed. So if the sprint time is 2 weeks, this team has a velocity of 20 story points per week. Before discussing estimation in software development further, lets discuss uncertainty.

Uncertainty Affects Accuracy

Let’s add a bit of uncertainty to our first example.

You walk from your current position to the nearest door.

Estimate how long it will take you to accomplish this task? Unless you have a satellite phone and are in the desert, your estimate is probably in units of minutes or seconds. What is the uncertainty with your estimate? Most likely your uncertainty is probably in the units of seconds.

Let’s try another estimate.

You walk from Seattle, Washington to Washington D.C.

image

Estimate how long will this journey take? Go ahead and really do this. I’ll wait.

What thought process did you go through. It was probably a bit different than the previous estimate. In this case, you probably needed to find out the distance (unless you are a geography wizard). Most people don’t possess the domain knowledge required to make this estimate without a bit of homework. No problem, a quick Google search tells it is about 2700 miles. Then you probably estimated your velocity. An average person’s walk is around 3 miles per hour. So that is about 900 hours of walking. Let’s say we walk 8 hours of the day. That puts us in Washington D.C. in about 113 days. What is the uncertainty in your estimate? Is it in units of hours? or days? Because of the size of this task, the initial uncertainty is probably in days (and maybe even weeks).

The estimate to the nearest door, it was too simple and you were probably were able to provide an estimate without the formal process of a distance and velocity approximation. In the second example, the size (or effort) of the problem (in terms of distance) was much larger. There was a lot more uncertainty in our distance and velocity approximations. The uncertainty in our estimate in the second case was much larger.

Now lets do one more estimation task. Imagine you are 100 days into your journey to Washington. Traveling 100 days has certainly provided a lot of experience. You probably have a much more accurate approximation to your velocity. You also have gained some confidence and understanding of your abilities. What has happened to your estimates? When will you arrive? What is the uncertainty?

What happens to your estimation and accuracy the closer you get? When the city limit sign for Washington D.C. becomes visible you could certainly provide an estimate with an accuracy in the minutes. There is a relationship between the accuracy and the amount of time remaining in the trip. Within steps of the finish line your uncertainty is in measured in seconds. While at the start of the trip it was probably in weeks.

From our thought experiments we can deduce the following ‘principles’:

  1. Estimates require domain knowledge.

  2. Tasks with a large size (or effort) are harder to estimate than smaller tasks.

  3. Uncertainty decreases as we gain confidence and understand our abilities.

  4. Uncertainty continually decreases as we get closer to  the finish.

So let’s apply this knowledge to estimation in our software projects.

Size The Work

First let’s spend some time discussing how we determine the size of the features we must estimate. We must learn from the past and not fall into the trap of the waterfall development process. Most development teams are moving towards a more agile or lean development methodology. We have to be careful in the process of sizing our features that we do not fully analyze, break down into tasks, and plan the development for all the features up front.

On the surface this can appear to be desirable. The problem is that a lot of energy (resources and time) go into the analysis, breakdown and planning. But the tasks we plan are subject to more uncertainty the farther in the future they will occur (principle #4). This uncertainty generally manifests itself in the form of change. Change to requirements, design, development, testing…etc. The can result in rework and in terms of lean development represents waste. One of the best discussions of ‘what went wrong’ with waterfall and why agile methodologies are better was in a video by Glenn Vanderburg at JRubyConf 2010.

Sizing of a task requires that the people with the domain knowledge be involved (principle #1). Be certain to include a cross-functional (requirements, designers, developers, testers) representatives. Minimize the sizing team for efficiency, but try to include a representative from all the domains on the team. If a story includes a specialized domain that is normally not present then size the story with the current team. Outside the sizing meeting query the domain expert to determine if they concur with the estimate. If the original estimate is not congruent with the expert’s, then cycle this story back through the sizing team.

Typically sizing is done using relative units and not hours. Sizing the work in terms of hours is more difficult and requires the higher level story to be broken down into tasks. Answering which is bigger (relative measurement) a Chihuahua or a German Sheppard is easier than answering how tall is a Chihuahua or German Sheppard in inches (absolute measurement). A common relative sizing scale that is used is the Fibonacci series (1, 2, 3, 5, 8, 13, 21, 34, etc.) or a derivative there of (1, 2, 3, 5, 8, 13, 20, 40, 100). Other sizing teams apply t-shirt sizes. The challenge with the t-shirt sizes is they are not numerical (you could always map them to numbers) and are more difficult to perform computations (velocity, burn down…etc).

Features should be broken down into smaller units of work. If the size of the unit of work is too large (e.g. greater than 50), an epic for example, then it introduces more uncertainty in the estimate (principle #2). To minimize the uncertainty, break these large unit of works into user stories with sizes less than 20 (rule of thumb). Once you are in that range, only break it down further if there is a logical reason for doing so.

image

A popular scrum activity for sizing is called planning poker. Essentially the team is issue cards (shown above). Individually, each team member estimates the story. The average of every team member’s size can be used as the final size. If there are outliers (everyone votes around 8 except one team member who votes 50) they should be discussed. This will generally result in an equal amount of over and under estimates.

The goal of the sizing team is to quickly move through the stories (thus the timer in the above image). If the story is too large or not clear, do not spend too much time on it. Make a note and handle it off-line. When all the stories have been sized, total up the numbers to determine the size of the release.

The key to sizing and our ability to measure velocity is to consistently size stories. In other words, a story sized today and the same story sized six months from now should result in the same numerical value. This consistency will result in less uncertainty in the final estimate.

Measure Your Velocity

Recall in our trip to Washington at the 100 day mark. We had a much better estimate of our velocity, more confidence in our ability and could better estimate our arrival time. This is true for our development teams also. The teams should be allowed to work through the stages of group development (forming – storming – norming – performing). This allows the team to gain confidence and understanding of its abilities (principle #3).

If we want to have a good approximation to our team’s velocity we also need to have metrics. If you don’t measure the velocity, then you will not be able to provide the average value or understand the uncertainty. All of the agile methodologies already have a baked in velocity concept.

In scrum the team iterates on time-boxed intervals called sprints.

image

During the sprint planning the team selects the highest priority stories from the product backlog. They select enough stories to fill their sprint backlog based upon the current estimate for their velocity (story points per sprint). When the sprint concludes, the team’s velocity is recalculated based upon the sprint just completed.

In pull-based methodologies (e.g. kanban), a cumulative flow diagram (shown below) is generally used to calculate the lead time.

image

The lead time is the time it takes a story point to propagate through the production pipeline. This is done by recording when a story enters the pipeline and when it exits. As stories exit, the lead time calculation is adjusted to provide a more accurate velocity for the team.

One very important point is to empower the team to have repeatable performance and thus a predictable velocity. Yes, this is a business and the team should be expected to perform. However, it can be detrimental to predictability to interfere with the team by changing team members, expecting the team to pull all nighters for a week to meet a deadline, or not enforcing work in process limits. These types of activities generally are not sustainable or undermine the performance of the team.

Dealing With Uncertainty

Now we have our best approximations to the sizes of the stories. Our team has been working together for awhile and are performing with a consistent velocity. Adding up all the story points and dividing by the velocity provides an estimate for the lead time for these features. This is our very best estimate. Even though we have done our very best, there is still some uncertainty in our estimate. What can we do?

First, realize that nothing is perfect. Striving for perfection can lead to paralysis by analysis. It is better to be professional about your estimates by discussing the uncertainty with the customer. Explain the sizing process, velocity and how estimates are calculated. End with an explanation of the ‘cone of uncertainty’ or the concept that uncertainty continually decreases as we near the finish (principle #4).

image

The key to dealing with this is communication with the customer. Many of the agile mythologies include structure to encourage this communication. As the software evolves, the customer should get frequent opportunities to inspect the product. Explain your development methodology. Describe and relay the importance of their role in the process. Just as the software evolves, so should our estimate. The estimate will become more accurate as we proceed. Customers are not surprised by the product, schedule or costs when professionals are involved. Constant communication is key.

The customer has the responsibility to prioritize the stories to ensure the most important stories get developed first. There may be exceptions with this related to retiring risky stories. Generally, the product should evolve with the customers highest priority requirements first.

There are uncertainties in our estimate, but there are also uncertainties in the customer’s expectations. Many times, during a customer inspection they will begin to speak about new features. Often seeing a prototype in action opens up creativity. This can lead to new user stories. The customer may even prioritize these new stories above the currently scoped work. Because this is common, a frank and professional discussion should happen at the start of the project with respect to scope changes. There may be financial, time, or other constraints that prevent the scope from expanding. To prevent disappointment customers must understand your scope change policy.

Summary

Remember estimation is not the same thing as guessing. As estimate is comprised of approximations that introduce uncertainty into our answer. Although we can do our best to minimize the uncertainty in our approximations, we simply cannot make it go away. Apply the ‘4 principles’ and deal with uncertainty in a professional manner. The key to providing good estimates is to collect and leverage metrics on the team. This post avoided the complexity of mathematical statistics. Adding a bit of statistical analysis to the metrics can provide valuable information. Providing good estimates can be hard. Hopefully, this discussion makes it a bit easier.

Posted in agile, best practice | 4 Comments

A .NET Generic Repository Pattern With Implementations

I previously blogged about the repository pattern and .NET implementations. Here are the links to those old posts:

Since that time, I have zeroed in on a fairly robust implementation of the repository pattern. I have used this implementation on a number of projects. The number of changes to the underlying interfaces and the NHibernate concrete implementation have slowed down. I find this generic repository implementation very useful and hope that you will also. I have posted the repository code on Github.

Github

For the sake of keeping all the documentation in one place, I will be posting additional info in the Github repository also. The repository pattern implementation abstracts away the details of the persistence and ORM layer. This has advantages and disadvantages. The reason I like the pattern is two fold:

  • Provides a consistent interface to the higher level layers of the application.
  • Allows the persistence layer to be faked for testing.

Example Usage

Here is an example usage:

// Create an instance of the session factory. This is typically done
//  once and cached. DbSessionFactory is a concrete implementation
//  of IDbSessionFactory using a variety of ORMs (e.g. NHibernate,
//  EF...etc).
//
IDbSessionFactory dbSessionFactory = new DbSessionFactory(connectionString)

// Create a session. This represents a database transaction.
//
using( IDbSesseion session = dbSessionFactory.Create())
{
    // Create a repository.
    //
    IKeyedRepository<Guid, Person> repo = session.CreateKeyedRepository<Guid, Person>();

    // Perform actions on the repository
    //
    Person person = new Person {Id = Guid.NewGuid(), FirstName = "Bob", LastName = "Cravens" };
    repo.Add(person);

    // Commit the transaction.
    //
    session.Commit();
}

Typically the IDbSessionFactory is created once at the start of the application. The concrete implementation of this interface would most likely use your favorite ORM on top of your favorite database. The Github repository currently provides two concrete implementations: NHibernate and an in-memory fake. You can use the fake for automated testing.

Conclusion

Visit the Github source and let me know if you see improvements or if you find this useful. I would like to add a few more implementation for some of the other popular ORMs. If you want to contribute an implementation please let me know.

Posted in .net, data access, design, nhibernate, repository | 2 Comments

Testing Your CRUD Using Generics

I often use a generic repository pattern as a wrapper around my data access logic. This wrapper generally serves the purpose of encapsulating specific implementation details. Here are the repository interfaces that I current use:

public interface IReadOnlyRepository<TEntity> where TEntity:class
{
    IQueryable<TEntity> All();
}
public interface IRepository<TEntity> : IReadOnlyRepository<TEntity> where TEntity:class
{
    bool Add(TEntity entity);
    bool Add(IEnumerable<TEntity> items);
    bool Update(TEntity entity);
    bool Delete(TEntity entity);
    bool Delete(IEnumerable<TEntity> entities);
}
public interface IKeyedRepository<TKey, TEntity> : IRepository<TEntity>
    where TEntity : class, IKeyed<TKey>
{
    TEntity FindBy(TKey id);
}
public interface IKeyed<TKey>
{
    TKey Id { get; }
}

 

I have a couple of previous posts on the repository pattern.

After creating a specific implementation of the repository, I often want to do some simple tests of the create, read, update and delete (CRUD) functions. This allows a quick test to ensure that my repository is configured correctly (DB configuration, entity mappings…etc). These tests are not meant to be part of my automated test bed. These are simply something that I run locally with a real database to get a quick “go” or “no go” indication. To accomplish these tests usually requires the following steps:

  1. Create an instance of the repository for the entity type.
  2. Get a count of the number of entities in the DB before we start.
  3. Instantiate an entity using random data.
  4. Add the entity to the DB.
  5. Read the entity back out of the DB.
  6. Compare and ensure they are equal.
  7. Modify the original entity.
  8. Update the entity in the DB.
  9. Read the modified entity from the DB.
  10. Compare and ensure the entity was modified.
  11. Delete the entity.
  12. Get a count of the number of entities in the DB after.
  13. Compare the count before and after and ensure they are equal.

This is not a full battery of tests, but hits the all the CRUD methods. I like to do this for each type of entity. This is very repetitive and provides a good opportunity to use Generics to simplify the process.

Create and Update Random Entities

The following bit of code generates an entity with properties that have been initialized with random values.

public static class Random<T> where T:class, new()
{
    private static readonly Random _random = new Random(DateTime.Now.Millisecond);

    public static T Create()
    {
        return Create(null);
    }

    public static T Create(List<string> protectedPropertyNames)
    {
        T result = new T();
        return Update(result, protectedPropertyNames);
    }

    public static T Update(T obj)
    {
        return Update(obj, null);
    }

    public static T Update(T obj, List<string> protectedPropertyNames)
    {
        Type type = typeof (T);
        foreach (PropertyInfo propertyInfo in type.GetProperties())
        {
            PropertyInfo info = propertyInfo;
            bool isFound = protectedPropertyNames == null ? false : protectedPropertyNames.Any(name => name.ToLower() == info.Name.ToLower());

            if (propertyInfo.CanWrite && !isFound)
            {
                if (propertyInfo.PropertyType == typeof(DateTime))
                {
                    propertyInfo.SetValue(obj, DateTime.Now, null);
                }
                else if (propertyInfo.PropertyType == typeof(int))
                {
                    propertyInfo.SetValue(obj, _random.Next(), null);
                }
                else if (propertyInfo.PropertyType == typeof(bool))
                {
                    propertyInfo.SetValue(obj, _random.Next(0, 2) == 0 ? false : true, null);
                }
                else if (propertyInfo.PropertyType == typeof(string))
                {
                    propertyInfo.SetValue(obj, Guid.NewGuid().ToString(), null);
                }
                else if (propertyInfo.PropertyType == typeof(double))
                {
                    propertyInfo.SetValue(obj, _random.NextDouble(), null);
                }
                else if (propertyInfo.PropertyType == typeof(byte))
                {
                    propertyInfo.SetValue(obj, byte.Parse(_random.Next(byte.MinValue, byte.MaxValue).ToString()), null);
                }
                else if(propertyInfo.PropertyType == typeof(Guid))
                {
                    propertyInfo.SetValue(obj, Guid.NewGuid(), null);
                }
            }
        }

        return obj;
    }
}

 

The code above serves two functions: create a new entity and update an existing entity. The create is really an update on a new entity and falls through to the same update code. The methods take an optional parameter that allows a list of properties to be provided that should not be updated. This allows you to set IDs that refer to existing objects without the reference being broken. The list of “if-else if” statements could be extended to include additional types.

A Generic CRUD Tester

Now that we can create and update entities with properties having random values. We need a generic test bench for the CRUD methods.

public static class CrudTester<T> where T: class, IKeyed<int>, new()
{
    public static void Test(ISessionFactory sessionFactory)
    {
        Test(sessionFactory, null);
    }

    public static void Test(ISessionFactory sessionFactory, List<string> protectedPropertyNames)
    {
        T entity1 = Random<T>.Create(protectedPropertyNames);
        Test(sessionFactory, protectedPropertyNames, entity1);
    }

    public static void Test(ISessionFactory sessionFactory, List<string> protectedPropertyNames, T seed)
    {
        int countBefore;

        using (UnitOfWork unitOfWork = new UnitOfWork(sessionFactory))
        {
            IKeyedRepository<int, T> repo = new Repository<int, T>(unitOfWork.Session);

            countBefore = repo.All().Count();

            // create
            repo.Add(seed);
            Assert.IsTrue(seed.Id > 0);

            unitOfWork.Commit();
        }

        T entity2;
        using (UnitOfWork unitOfWork = new UnitOfWork(sessionFactory))
        {
            IKeyedRepository<int, T> repo = new Repository<int, T>(unitOfWork.Session);

            // read
            entity2 = repo.FindBy(seed.Id);
            Assert.IsTrue(seed.Equals(entity2));

            unitOfWork.Commit();
        }

        using (UnitOfWork unitOfWork = new UnitOfWork(sessionFactory))
        {
            IKeyedRepository<int, T> repo = new Repository<int, T>(unitOfWork.Session);

            // update
            seed = Random<T>.Update(seed, protectedPropertyNames);
            repo.Update(seed);
            T entity3 = repo.FindBy(seed.Id);
            Assert.IsFalse(entity2.Equals(seed));
            Assert.IsTrue(entity3.Equals(seed));

            unitOfWork.Commit();
        }

        using (UnitOfWork unitOfWork = new UnitOfWork(sessionFactory))
        {
            IKeyedRepository<int, T> repo = new Repository<int, T>(unitOfWork.Session);

            // delete
            repo.Delete(seed);
            int countAfter = repo.All().Count();
            Assert.IsTrue(countAfter == countBefore);

            unitOfWork.Commit();
        }
    }
}

 

This above code provides three entry points with various levels of control over the test. The following parameters are either specified externally or internally generated:

  • ISessionFactory – An NHibernate session factory (creates instances of ISession). This would need to be modified to work with other ORMs.
  • Protected Property Names – This provides a list of property names that should not be modified.
  • Seed – This provides a seed entity for testing allowing you to hook up your foreign keys and such.

All three entry points fall into the final ‘Test’ method. This method simply runs the CRUD tests that we previously outlined.

Summary

The above code provides an easy framework to create tests that look like the following:

public Crud_Test_User_Entity()
{
    NHibernateHelper helper = new NHibernateHelper(_connectionString);
    CrudTester<User>.Test(helper.SessionFactory);
}

public Crud_Test_Address_Entity()
{
    Address address = Random<Address>.Create();
    address.UserId = 10;  // assume this user exists in DB

    NHibernateHelper helper = new NHibernateHelper(_connectionString);
    CrudTester<Address>.Test(helper.SessionFactory, new List<string>{"UserId"}, address);
}

This allows the basic CRUD methods to quickly be tested to ensure the ORM is configured correctly.

Posted in data access, nhibernate, testing | Leave a comment

Upload Multiple Files With Progress Using Uploadify

I previously posted on how to use Uploadify to upload files. The previous post covered really well how to upload single files to an ASP.NET MVC controller using Uploadify. This post will expand on that a bit. First we will revisit how to upload multiple files using a single HTML input element. Then the case of uploading multiple files using multiple HTML input elements. Finally, how to upload multiple files while posting form data.

image

Here are links to the demo pages for the Single and Multiple file uploads.

Multiple  Single

Getting Started

If you are not familiar with Uploadify here is a description from their web site:

Uploadify is a jQuery plugin that integrates a fully-customizable multiple file upload utility on your website. It uses a mixture of JavaScript, ActionScript, and any server-side language to dynamically create an instance over any DOM element on a page.

If will first need to download the Uploadify code. I simply unzipped it into my ‘scripts’ folder as shown below:

image

Any page that uses Uploadify will need to include the following lines in the ‘head’ section:

<link href="<%= Url.Content("~/Content/Scripts/uploadify/uploadify.css") %>" rel="stylesheet" type="text/css" />

<script type="text/javascript" src="/demos/_common/scripts/jquery-1.4.1.js"></script>
<script type="text/javascript" src="<%= Url.Content("~/Content/Scripts/uploadify/swfobject.js") %>"></script>
<script type="text/javascript" src="<%= Url.Content("~/Content/Scripts/uploadify/jquery.uploadify.v2.1.4.min.js") %>"></script>

The first line links in the CSS used by Uploadify. The last three lines add the required JavaScript.

Multiple Files With A Single HTML Input

Uploadify directly supports uploading multiple files by configuring the Uploadify object. Add the following HTML input elements to your page:

<p><input type="file" id="multipleFiles" /></p>
<p><button id="btn1">Upload Files</button></p>

The ‘input’ element will be used as the Uploadify object. The ‘button’ element serves as a trigger to start the uploads. Here is the JavaScript that configures this scenario:

// Multiple files - single input
$("#multipleFiles").uploadify({
    'uploader': '<%= Url.Content("~/Content/Scripts/uploadify/uploadify.swf") %>',
    'script': '<%= Url.Action("Upload") %>',
    'fileDataName': 'file',
    'buttonText': 'File Input...',
    'multi': true,
    'sizeLimit': 1048576,
    'simUploadLimit': 2,
    'cancelImg': '<%= Url.Content("~/Content/Scripts/uploadify/cancel.png") %>',
    'auto': false,
    'onError': function (a, b, c, d) {
        if (d.status == 404)
            alert("Could not find upload script. Use a path relative to: " + "<?= getcwd() ?>");
        else if (d.type === "HTTP")
            alert("error " + d.type + ": " + d.status);
        else if (d.type === "File Size")
            alert(c.name + " " + d.type + " Limit: " + Math.round(d.info / (1024 * 1024)) + "MB");
        else
            alert("error " + d.type + ": " + d.text);
    },
    'onComplete': function (event, queueId, fileObj, response, data) {
        alert(response);
    }
});
$("#btn1").click(function () {
    $('#multipleFiles').uploadifyUpload();
});

The Uploadify object supports a number of properties, events, and methods. Visit their page for in-depth information. The configuration that allows multiple files to be selected is the ‘multi’ option. Setting this to ‘true’ enables multiple file selection as shown below.

image

That results in the following:

image

The ‘Upload Files’ button is wired up to call the ‘uploadifyUpload’ method on the Uploadify object. This begins the upload process. The file uploads happen asynchronously and not in a guaranteed to arrive in a specified order. All selected files are handled by a single ASP.NET MVC action method:

public string Upload(HttpPostedFileBase file)
{
    // Process the file here.
    //
    return "Upload processed. filename=" + file.FileName;
}

This works great if the same processing can be applied to all files. If you require the files to be dispatched out to different processors based upon their type then this probably will not work. In this case, you will need to use multiple HTML input elements.

Multiple Files With Multiple HTML Input Elements

Now we have multiple HTML ‘input’ elements on the page.

<p><input type="file" id="file1" name="file1" class="uploadify1" /></p>
<p><input type="file" id="file2" name="file2" class="uploadify1" /></p>
<p><input type="file" id="file3" name="file3" class="uploadify1" /></p>
<p><button id="btn2">Upload Files</button></p>

The ‘button’ element is again used to start the upload processing. Each ‘input’ element has a unique ‘name’ attribute. If this were posted back in a normal HTML ‘form’ then this ‘name’ attribute would follow the file to the server where the file processing could be based upon the value of the ‘name’ attribute. Here is the JavaScript that configures this scenario:

// Multiple files - multiple inputs
var packageId = 'multiple1' + new Date().getTime() + Math.round(Math.random() * 1000);
$(".uploadify1").each(function () {
    $(this).uploadify({
        'uploader': '<%= Url.Content("~/Content/Scripts/uploadify/uploadify.swf") %>',
        'script': '<%= Url.Action("Upload2") %>',
        'fileDataName': 'file',
        'buttonText': 'File Input...',
        'multi': false,
        'scriptData': { 'packageId': packageId, 'type': $(this).attr('name') },
        'sizeLimit': 1048576,
        'simUploadLimit': 1,
        'cancelImg': '<%= Url.Content("~/Content/Scripts/uploadify/cancel.png") %>',
        'auto': false,
        'onError': function (a, b, c, d) {
            if (d.status == 404)
                alert("Could not find upload script. Use a path relative to: " + "<?= getcwd() ?>");
            else if (d.type === "HTTP")
                alert("error " + d.type + ": " + d.status);
            else if (d.type === "File Size")
                alert(c.name + " " + d.type + " Limit: " + Math.round(d.info / (1024 * 1024)) + "MB");
            else
                alert("error " + d.type + ": " + d.text);
        },
        'onComplete': function (event, queueId, fileObj, response, data) {
            alert(response);
        }
    });
});
$("#btn2").click(function () {
    $(".uploadify1").uploadifyUpload();
});

In this case, each input element is configured to only allow a single file to be selected (‘multi’:false). The key configuration in this case is the ‘scriptData’ option. This options allows JSON data to be sent back to the server along with the file. In this case we send a ‘packageId’ (created on the first line) and a ‘type’. The ‘packageId’ is used to associate the files into groups on the server. The ‘type’ variable is the ‘input’ element’s ‘name’ attribute and allows the server side code to differentiate the files. The ASP.NET MVC controller now looks like the following:

public string Upload2(HttpPostedFileBase file, string packageId, string type)
{
    // Add the file to the package.
    //
    Package package = PackageManager.GetPackage(packageId);
    package.AddFile(file, type);
    return "Upload received.\nFilename: " + file.FileName + "\nPackage Id: " + packageId + "\nType: " + type;
}

This method not only receives the file, but also the JSON data sent by Uploadify. The JSON data is then used to add the incoming file to a particular package. The specific implementation of the packaging system is not important (and not shown), but the key is that the server has the information necessary to implement the packaging feature. The above code could just as easily dispatched the file to various processors based upon the ‘type’ parameter.

The above works great for processing files based upon their types. The next challenge is processing files and form data where the form data provides important context for processing.

Multiple Files With Multiple HTML Input Plus Form Data

Assume we have an HTML ‘form’ that accepts multiple files and the first/last name of the user. The files must be processed based upon who uploaded the files. Therefore the first/last name data is important and required before processing can occur. Here is the HTML defining the form:

<form id='form1' method='post' action='<%= Url.Action("ProcessForm") %>'>
    <p><em>Provide your name:</em></p>
    <p>First Name <input type="text" name="firstName" id="fname" /></p>
    <p>Last Name <input type="text" name="lastName" id="lname" /></p>
    <p><em>Using the buttons below, navigate and select up to three files (1MB size limit) then click the 'Submit' button:</em></p>
    <p><input type="file" id="file4" name="file1" class="uploadify2" /></p>
    <p><input type="file" id="file5" name="file2" class="uploadify2" /></p>
    <p><input type="file" id="file6" name="file3" class="uploadify2" /></p>
    <p><input type="submit" id="btn3" /></p>
</form>

Traditionally (without Uploadify involved) this form would POST all the data to a single ASP.NET MVC action method and no upload progress would be provided. Hooking up Uploadify provides the desired progress indicators, but now the files are uploaded ‘out of band’. Some special handling is necessary to keep the files and the form data associated. Here is the JavaScript that configures the form:

// Multiple files - multiple inputs in a form
var packageId2 = 'multiple2' + new Date().getTime() + Math.round(Math.random() * 1000);
var elem = $("<input type='hidden' name='packageId' value='" + packageId2 + "' />");
$("#form1").prepend(elem);
$(".uploadify2").each(function () {
    $(this).uploadify({
        'uploader': '<%= Url.Content("~/Content/Scripts/uploadify/uploadify.swf") %>',
        'script': '<%= Url.Action("Upload2") %>',
        'fileDataName': 'file',
        'buttonText': 'File Input...',
        'multi': false,
        'scriptData': { 'packageId': packageId2, 'type': $(this).attr('name') },
        'sizeLimit': 1048576,
        'simUploadLimit': 1,
        'cancelImg': '<%= Url.Content("~/Content/Scripts/uploadify/cancel.png") %>',
        'auto': false,
        'onError': function (a, b, c, d) {
            if (d.status == 404)
                alert("Could not find upload script. Use a path relative to: " + "<?= getcwd() ?>");
            else if (d.type === "HTTP")
                alert("error " + d.type + ": " + d.status);
            else if (d.type === "File Size")
                alert(c.name + " " + d.type + " Limit: " + Math.round(d.info / (1024 * 1024)) + "MB");
            else
                alert("error " + d.type + ": " + d.text);
        },
        'onComplete': function (event, queueId, fileObj, response, data) {
            incrementUploadCount();
        }
    });
});
$("#btn3").click(function (evt) {
    evt.preventDefault();

    submit();
});

The Uploadify elements are configured nearly the same as before using the ‘scriptData’ option to send back the ‘packageId’ and the ‘type’ information as JSON. However, in this case the ‘packageId’ is prepended to the form as an HTML hidden ‘input’ element (first three lines). In addition the Uploadify ‘onComplete’ callback now calls the ‘incrementUploadCount’ function (covered below) and the HTML submit button is wired up to call the ‘submit’ function.

var numFilesSelected = 0;
var numFilesUploaded = 0;
function submit() {
    // validate the form
    if ($('#fname').val() === '') {
        alert('Provide a first name.');
        return;
    }
    if ($('#lname').val() === '') {
        alert('Provide a last name.');
        return;
    }

    // determine the number of files that need to be
    //  uploaded and reset the uploaded count
    numFilesSelected = $(".uploadifyQueueItem").length;
    numFilesUploaded = 0;
    // upload the files
    $('.uploadify2').uploadifyUpload();
}

function incrementUploadCount() {
    numFilesUploaded++;
    if (numFilesUploaded === numFilesSelected) {
        alert('About to submit the form data.');
        postFormData();
    }
}

function postFormData() {
    $("#form1").submit();
}

The above code defines the client side pipeline that eventually upload the files and submit the form. The submit button calls the ‘submit’ function. In the ‘submit’ function, the form is first validated. Then the number of files being uploaded is determined and a counter is set to zero. Finally, the Uploadify elements are triggered.

Each Uploadify element calls the ‘incrementUploadCount’ upon successfully uploading the file to the server. Once all the files have been uploaded, the form is submitted. Here is the ASP.NET MVC action methods that are needed:

public string Upload2(HttpPostedFileBase file, string packageId, string type)
{
    // Add the file to the package.
    //
    Package package = PackageManager.GetPackage(packageId);
    package.AddFile(file, type);
    return "Upload received.\nFilename: " + file.FileName + "\nPackage Id: " + packageId + "\nType: " + type;
}

[AcceptVerbs(HttpVerbs.Post)]
public ActionResult ProcessForm(string firstName, string lastName, string packageId)
{
    Package package = PackageManager.GetPackage(packageId);
    package.FirstName = firstName;
    package.LastName = lastName;
    PackageManager.RemovePackage(package.Id);
    return View("UploadReceipt", package);
}

The ‘Upload2’ action method is the same as before. It collects the uploaded files into a package. The ‘ProcessForm’ action method receives the rest of the form data along with the ‘packageId’. The ‘packageId’ is used to request the ‘Package’ that contains the associated files. The form data is then added to the ‘Package’. Once complete the ‘Package’ can be sent off for processing as a whole. In this case, it is simply sent to a ‘View’ to render a receipt page.

Summary

Uploading multiple files with progress using Uploadify is an option. Depending on your requirements there can be additional complexity. In this post, we have covered a way of accomplishing multiple file uploads under various requirements. As always, if you have feedback post a comment.

[ Update ]

Here is the package manager that I put together. I did not put a lot of thought or testing into this implementation.

public class UploadedFile
{
    public HttpPostedFileBase Stream { get; set; }
    public string Type { get; set; }
}

public class Package
{
    public string Id { get; private set; }
    public List<UploadedFile> Files { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }

    public Package(string id)
    {
        Id = id;
        Files = new List<UploadedFile>();
    }

    public void AddFile(HttpPostedFileBase file, string type)
    {
        UploadedFile fileInfo = new UploadedFile { Stream = file, Type = type };
        Files.Add(fileInfo);
    }
}

public static class PackageManager
{
    private static readonly List<Package> _packages = new List<Package>();

    public static Package GetPackage(string packageId)
    {
        Package package = _packages.Where(p => p.Id == packageId).SingleOrDefault();
        if (package == null)
        {
            package = new Package(packageId);
            _packages.Add(package);
        }
        return package;
    }

    public static void RemovePackage(string packageId)
    {
        Package package = _packages.Where(p => p.Id == packageId).SingleOrDefault();
        if (package != null)
        {
            _packages.Remove(package);
        }
    }
}
Posted in asp.net, jQuery, mvc, web | 20 Comments

A Modal Dialog Plugin for jQuery

There are a number of great jQuery modal plug-ins that already exist. So, why am I cluttering the interwebs with yet another one? There are a number of reasons:

  • form[1]Feature Bloat – A number of plug-ins offer a lot of features. At times so many features that the plug-in became too complicated to use. I want the plug-in to be simple to integrate into my page.
  • Styling Issues – Some modal plug-ins do not separate the CSS for visual styling (e.g. border and background-color) the modal from those styles necessary to create a modal (e.g. positioning and z-index). I want the visual styling to be 100% in my page’s CSS.
  • Learning Experience – This is probably the most important reason (at least to me). Writing code, using your creations and sharing them with the community are three ‘bullet-proof’ ways to improve your skills.

For more information or to download check out the demo page!

Demo

Posted in jQuery | Leave a comment

Loosely Coupled JavaScript Using PubSub

I recently watched a very interesting video by Rebecca Murphy that discussed using the pubsub architecture to create loosely coupled JavaScript. The concept is to develop your JavaScript objects and allow the communication between the objects to occur via events that are managed by a publisher / subscriber (pubsub) service. The diagram below illustrates a pubsub service that exposes three methods: publish, subscribe, and unsubscribe. In this case, two ‘panels’ are subscribing to the service.

image A data source in the system then publishes data to the pubsub service. The pubsub service forwards this data to the appropriate subscribers. This is illustrated below.

imageThe pubsub service allows subscriptions to specific ‘events’. Much like you can subscribe to specific channels with your cable TV provider.

Let’s take a look at a specific example. Imagine we have a web page. On that we page is a form that allows the user to enter their name, lucky number, and select a favorite color.

imageIn addition to this form we have a number of other panels that present the data in a specialized way. For example we could have a panel that presents the names of all registered users. Another panel could present data about only the most recent users. While another displays a list of unique lucky numbers. Yet another presents color swatches of all the favorite colors. The development of these panels could be quite complex. Each will probably have its own HTML, CSS, and JavaScript.

image

By using the pubsub service to couple these panels to the data source, each can be developed independently of the others. The development process feels a bit like using an IOC container. Develop each component and then bind them together. Here is a link to a demo page.

Demo

HTML

The markup for this example is shown below:

<div id="page">
    <div id="form">
        <p>
            <label for="first">First Name</label>
            <input type="text" id="first" />
        </p>
        <p>
            <label for="last">Last Name</label>
            <input type="text" id="last" />
        </p>
        <p>
            <label for="number">Lucky Number</label>
            <input type="text" id="number" />
        </p>
        <p>
            <label for="color">Favorite Color</label>
            <input type="text" id="color" name="color" value="#006699" />
            <div id="colorpicker"></div>
        </p>
        <p>
            <button id="add">Add</button>
        </p>
    </div>

    <div id="panel-wrap">
        <div id="names" class="panel"></div>
        <div id="lastuser" class="panel"></div>
        <div id="numbers" class="panel"></div>
        <div id="colors" class="panel"></div>
    </div>

</div>

There are two main sections. The top defines the form (in a literal sense not the html element) that is used as the data source. The bottom creates placeholders for the panels.

JavaScript

I am using a slightly modified version of a pubsub plug-in created by Peter Higgins. The only modification that I did was to add a try catch block around the publish method. Without that modification any one subscriber can prevent later subscribers from receiving the event by having ‘exceptional code’ (in a bad way). For completeness here is the final code (without the comments):

;(function(d){
    var cache = {};

    d.publish = function(/* String */topic, /* Array? */args){
        try{
            d.each(cache[topic], function(){
                this.apply(d, args || []);
            });
        } catch (err) {
            // handle this error
            console.log(err);
        }
    };

    d.subscribe = function(/* String */topic, /* Function */callback){
        if(!cache[topic]){
            cache[topic] = [];
        }
        cache[topic].push(callback);
        return [topic, callback]; // Array
    };

    d.unsubscribe = function(/* Array */handle){
        var t = handle[0];
        cache[t] && d.each(cache[t], function(idx){
            if(this == handle[1]){
                cache[t].splice(idx, 1);
            }
        });
    };

})(jQuery);

 

The main page uses this service to bind publishers and subscribers to each other. The relevant JavaScript is shown below:

// Initialize the panels
//    1st param: the container element
//    2nd param: the event to subscribe
var userPanel = new UserPanel(      $('#names'),    'new/user');
var lastPanel = new LastUser(       $('#lastuser'), 'new/all');
var numberPanel = new LuckyNumbers( $('#numbers'),  'new/number');
var colorPanel = new FavColors(     $('#colors'),   'new/color');

// Bind to the click event
$('#add').click(function(){

    // Get the values from the form
    var first = $('#first').val();
    var last = $('#last').val();
    var color = $('#color').val();
    var number = $('#number').val();
    if(isNaN(number)){
        alert("woah...that's not a number");
        $('#number').focus();
        return;
    }

    // Publish the new data
    $.publish('new/user',   [first,last]);
    $.publish('new/number', [number]);
    $.publish('new/color',  [color]);
    $.publish('new/all',    [first,last,number,color]);
});

First this code creates and initializes four panel objects. Each panel takes a container element and an event in the constructor. Each panel then subscribes to the appropriate event and renders itself. The code above binds to the ‘click’ event of the form that we created earlier. At the bottom of that binding are the publish events supported by the form. The data is sliced up and made available via a number of events that expose only that slice.

Each panel is contained in its own JavaScript file. They all follow the same pattern. The following code shows the ‘UserPanel’ object. View source the demo for the other panels.

function UserPanel(parent, event){
    var _parent = parent;
    var _html = '';
    var _title = 'Registered Users';
    var _tmpl = '<li>{{name}}</li>';

    var _renderUserPanel = function(name){
        if(name!=undefined){
            var elem = _tmpl.replace('{{name}}', name);
            _html = _html + elem;
        }

        var result = '<h3>' + _title + '</h3>';
        if(_html!=''){
            result += '<ul>' + _html + '</ul>';
        } else {
            result += '<ul><li>no users</li></ul>';
        }
        _parent.html(result);
    }

    var _subscribe = function(event){
        $.subscribe(event, function(first, last){
            var name = first + ' ' + last;
            _renderUserPanel(name);
        });
    }

    // render the initial markup
    _renderUserPanel();
    if(event!=undefined){
        _subscribe(event);
    }
}

Summary

There is a definite trade off with this architecture. On one hand there is a bit more code. On the other hand the architecture is more decoupled. I struggled a bit to come up with a meaningful example that tips the balance in favor of decoupled architecture. With simple pubsub examples, the added code appears as too much cost for the added benefit. I hope this example has found the right balance. If you know of a better example, please provide links in the comments.

Posted in jQuery, javascript | 7 Comments

Web Charts Using jQuery Flot

Adding charts to your web application is simple using a library called ‘Flot’. Flot is a pure JavaScript plotting library for jQuery. It produces graphical plots of arbitrary datasets on-the-fly client side. During this post I will explore using the Flot library a bit.

image image

Basic Charting

First head on over to the Flot project web site and download the Flot code. Save it to a file called ‘jquery.flot.js’. Then simply add a couple of ‘script’ tags to link the library into your web page.

<!--[if IE ]>
<script type="text/javascript" src="http://bobcravens.com/files/excanvas.min.js"></script>
<![endif]-->
<script type="text/javascript" src="http://bobcravens.com/files/jquery.flot.js"></script>

Notice for IE we are including the Explorer Canvas JavaScript emulator. Next add a place holder ‘div’ element somewhere in the body element of your html.

<div id='chart1' style='width:400px;height:300px;'></div>

Fetch (or in this case create) and format some data to plot. Flot uses the following format for data:

var series1 = [ [0,0], [1,1], [2,2], [3,3], [4,4] ];
var series2 = [ [0,4], [1,3], [2,2], [3,1], [4,0] ];

var data = [ series1, series2 ];

Each point is an array of x and y values. A series is an array of points. The main method that creates plots accepts an array of series as input values. Finally create the plot using the following call:

$.plot($("#chart1"), data);

The Flot library has a good set of default for many visual elements. The above is the ‘bare-bones’ graph. Create some data, provide a place holder and create the plot.

Demo 1

Formatting Example

If you want a more control over the formatting Flot gives you plenty of hooks. Previously we discussed a series being an array of points. Flot actually allows a series to be a JavaScript object with properties. As an example the following code formats the label, line type, and color for each of the series:

var series1 = {
            label: "series 1",
            data: [ [0,0], [1,1], [2,2], [3,3], [4,4] ],
            color: "#ff0000",
            points: { show: false },
            lines: { show: false },
            bars: { show: true },
            yaxis: 1
            };
var series2 = {
            label: "series 2",
            data: [ [0,4], [1,3], [2,2], [3,1], [4,0] ],
            color: "#00ff00",
            points: { show: true },
            lines: { show: true },
            yaxis: 2
            };

There are many other options that exist. Flot also allows global configurations to be set by passing a third parameter into the ‘plot’ method.

var options = {
    legend: { show: true, position: "nw" }
            };

$.plot($("#chart2"), data, options);

In the above case, the legend is positioned in the top-left (north-west) corner of the chart.

Demo 2

Hover / Click Example

Interacting with the charts is also simple with Flot. To enable the mouse events use the following options:

var options = {
    legend: { show: true, position: "nw" },
    grid: { hoverable: true, clickable: true }
};

The ‘grid’ object above has two flags set to true that enable hover and click events. Your code can bind to these through the ‘plothover’ and ‘plotclick’ events.

$("#chart3").bind("plothover", function(event, pos, item){
    $("#hover").html("hover: x=" + pos.x.toFixed(2) + ", y1=" + pos.y.toFixed(2) + " , y2=" +  pos.y2.toFixed(2));
    if(item){
        $("#item").html("item: series=" + (item.seriesIndex + 1) + ", x=" + item.datapoint[0].toFixed(2) + ", y=" + item.datapoint[1].toFixed(2));
    }
});

$("#chart3").bind("plotclick", function(event, pos, item){
    if(item){
        msg = "item: series=" + (item.seriesIndex + 1) + ", x=" + item.datapoint[0].toFixed(2) + ", y=" + item.datapoint[1].toFixed(2);
        alert(msg);
    }
});

Both of these bindings provide three arguments to the callback function: event, pos, item.  The following is a screenshot of the console log output in firebug for those parameters.

image

The ‘plothover’ event uses this data to update two div elements (‘#hover’ & ‘#item’).

Demo 3

Dynamic Data

Flot can also be used to plot dynamically changing data. This data could will probably originate on the server and pulled to the client via an AJAX call. Here is a trivial example demonstrating how to update the chart data.

var xVal = 0;
var data = [[],[]];
var plot = $.plot( $("#chart4"), data);

function getData(){
    // This could be an ajax call back.
    var yVal1 = Math.floor(Math.random()*11);
    var yVal2 = Math.floor(Math.random()*11);
    var datum1 = [xVal, yVal1];
    var datum2 = [xVal, yVal2];
    data[0].push(datum1);
    data[1].push(datum2);
    if(data[0].length>10){
        // only allow ten points
        data[0] = data[0].splice(1);
        data[1] = data[1].splice(1);
    }
    xVal++;
    plot.setData(data);
    plot.setupGrid();
    plot.draw();
}

setInterval(getData, 1000);

In this example the ‘getData’ function would probably generate an AJAX call with a callback that does the chart update. Here, I am using the JavaScript Math functions to generate random data. This code also only keeps the 10 most recent points using the ‘splice’ function. Calling ‘setData’ on the plot object updates the data. A call to ‘setupGrid’ recalculates the grid. Finally the call to ‘draw’ re-renders the data.

Demo 4

Summary

The Flot library is very powerful and easy to use. Check out the other examples supplied by the Flot team.

Posted in jQuery, javascript, web | 41 Comments