Today I want to tell you a little more about how and why we are doing this.

Why?

For the longest time, our tests and builds were a mystery to our users. They all happened behind closed doors and nobody really knew what we were doing. Over the last few years, we started adding more and more of our tests to our source tree, so that people can see what we are actively testing. This helped a little, but it didn’t show that we run every test on every build and that if a test fails, we don’t push that build to GitHub.

Then along came Travis CI, the open source continuous integration platform that hooks directly into GitHub. I played around with it for a day and immediately began moving our tests around so we could use it. We want to be more transparent in our processes and allow the public to see what we test and how we do it.

One thing we want to add to this process is access to our current code coverage report. Currently, we run a code coverage report daily from tests executed in FF12, Chrome-latest and IE8, but none of our users can see that we are at ~80% covered:

We are working very hard to rectify this issue and hopefully get this data out in the open for everyone to see.

How?

I had to make minimal changes to our core test suites in order to get them to run under Node.js in Travis CI. By “core test suites”, I mean any YUI module that can execute without the need for a working DOM. This includes, but is not limited to: YUI Core, Loader, YQL, Y.Array, Y.Object, etc. These modules are all perfectly usable inside of Node.js without modification.

Let’s look at the YQL tests as an example.

All I had to do was create a Node.js wrapper similar to the standard test wrapper we use in a browser and include the exact same test that’s executed in the browser.

Here is the YQL module’s wrapper:

Now, this same test module can be executed in a browser and in Node.js without modification!

Caveats?

In order for these tests to run natively in Node.js, they need to not interact with the DOM. For example, Y.Array consists of helper methods for dealing with Arrays in JavaScript. But there are a few tests that include dealing with DOM elements to ensure that the helper methods return the right values. For these tests, I had to move the DOM-related code into a new test and add that test to the list of ignored tests when Y.UA.nodejs is detected. This way, such tests are ignored in Node.js, but still run in the browser.

Here’s an example:

As you can see, it’s relatively easy to make our tests run in both environments to ensure that our code is stable and fully functional when used in Node.js as it is in the browser.

What is Grover?

Grover is a command-line tool that allows you to execute YUITest-based tests in PhantomJS. PhantomJS is a headless Webkit instance that allows you to render an HTML page without a GUI present. So Grover closes the gap on this and allows you to run our unit tests from the command-line inside of a CI system like Travis.

Grover is free and available via: npm install -g grover (You must have the PhantomJS binary installed before using Grover.)

How do we see all this?

Travis CI provides a full report of previous builds, as well as an up-to-date status information. Below are the links to our current projects hosted on Travis CI:

• YUI 3
• YUIDoc
• Yeti
• Grover

Here is a small snippet of what our build output looks like (from the build history for the yui3 repo):

What else does it do?

We are trying out the new Travis/GitHub Pull Request feature on all of our projects. This means that whenever a developer submits a Pull Request to us, Travis will automatically pull their code, merge it into master (on their server) and run our full unit test suite against it. Their “Travis Bot” will automatically post a comment back to the Pull Request telling the developer whether or not their patch passed its tests.

Here’s an example of a Pull Request passing:

And one where it fails:

What’s next

We plan on adding support for executing our tests with Yeti as soon as it becomes stable enough to run on each build. We are also looking into deploying our code coverage numbers as well. Other than that, feel free to tell us in what other ways we can be more open than we are now.

I, for one, am very happy with all of these new features and I hope you are too!

Here are the target dates for all YUI 3.6.0 milestones:

• PR1 – May 8, 2012
• PR2 – June 12, 2012
• PR3 – July 17, 2012
• GA – July 31, 2012

An overview of items we are working on for YUI 3.6.0 include:

• ScrollView enhancements
• DataTable features and performance improvements
• Charts bug fixes
• Image preview feature in Uploader
• Popup Calendar
• YUI seed and global refactored for performance

We are also taking steps to condense our development cycle so that we can transition to monthly releases in the near future.

The list of issues addressed in 3.6.0 PR1 and the full list of issues targeted for 3.6.0 are available in our bug tracker.

We invite you to join the ongoing development discussions happening with the team over on GitHub.

We look forward to your continued feedback!

[View Larger Image]

YUIDoc provides:

• Live previews. YUIDoc includes a standalone doc server, making it trivial to preview your docs as you write.
• Modern markup. YUIDoc’s generated documentation is an attractive, functional web application with real URLs and graceful fallbacks for spiders and other agents that can’t run JavaScript.
• Wide language support. YUIDoc was originally designed for the YUI project, but it is not tied to any particular library or programming language. You can use it with any language that supports /* */ comment blocks.

Some of the new features added to this version are:

• Markdown support in code comments
• Support for many more tags out of the box
• Logic separated to allow for easy extensibility
• Better theming support
• Server mode for development time previews
• External data mixing
• Easy cross platform installation
• Cross-linking inside and out of current project
• JSON based configuration

Let’s get into a little more detail on some of these:

Simple Installation

If you have Node.js and NPM installed, installation is easy:

npm -g install yuidocjs

Markdown support in code comments

YUIDoc will parse your comment with Markdown before it applies the Handlebars template giving you great flexibility when writing your docs.

Logic separated to allow for easy extensibility

YUIDoc uses YUI’s class infrastructure internally and exports all of these modules when you require the yuidocjs module. This allows end users to hook into YUIDoc’s internals and change the way it does things. You can extend classes, augment them or just flat out change methods to suite your needs.

Better theming support

In this release we use the built-in Y.Handlebars helper to handle all template generation. We have also taken development into consideration when building this feature. YUIDoc will first search it’s built in theme directory for partials, then it will search your local theme directory. This allows you to only have to maintain the files you wish to change in your theme and not have to copy every partial even if you are not modifying it.

Server mode for development time previews

This is my favorite new feature! You can fire up YUIDoc in server mode and it will give you live previews of your documentation as you edit it. Simply save your file and reload the page from the built in server and see your changes live. Including external data and cross-linking. You no longer have to generate the docs for your entire project just to see a documentation change!

External data mixing

YUIDoc now allows you to link your documentation to the rendered output from another YUIDoc instance. For example, if your project is using YUI and extending some of our core classes, you can link to our exported data.json file (from our YUIDoc build) and when YUIDoc parses your documentation it will fetch our data and cross-link all of your extended classes back to ours. This way you don’t have to document another projects code, you simply point over to their docs like it was part of yours.

Project Changes

All future YUIDoc development will be fully conducted on Github. We will be tracking the project on their wiki and using their issues to manage our tickets. It will be run like a native Node.js project completely in the open. We will also be using a Google Group for support requests, so sign up today!

We are also happy to report that YUIDoc’s unit tests are hosted on Travis-CI and will run per Github push!

• Source
• Documentation
• API Docs
• Google Group
• Travis-CI

What about the old version of YUIDoc?

The old Python source for YUIDoc is in a branch on the current Github repo where it will remain indefinately. There are no plans on accepting any pull requests or making any updates to that code base.

Uploader was another component that saw a major upgrade in 3.5.0. Of the many changes, one big one was the addition of support for native features in browsers supporting HTML5.

Join us for a discussion with its author, Allen Rabinovich, in which we’ll talk about how Uploader is structured, some of the challenges with modern techniques that needed to be addressed, and what’s on the roadmap for upcoming versions.

Time & Details

We’ll be online on Thursday from 10am to 11am PDT.

This week, we’ll be trying out Google Plus Hangouts on Air, so the audio and video feed will be through G+, and we’ll be monitoring chat in the #yui IRC channel on freenode.

View the Recording

Enjoy!

The most straightforward way to edit data would be to use inline editing, but this is not yet available in YUI 3 DataTable. In the mean time, one option is to roll your own input fields directly in the table cells, similar to this example.

If you do not want to save each change when it happens, you could use the QuickEdit gallery module instead. This lets you edit all the visible cells and then save the changes in a single operation.

The above solutions are modal. If you prefer a non-modal way to edit all the cells, you could use the Bulk Editor widget. This works with pagination, so you can save all the changes across all the pages of your table in a single operation. It even lets you easily add and remove rows — and even when you are using a remote DataSource!

Bulk Editor is not built on top of YUI DataTable, however. If you need other features from YUI DataTable, you could use the state preservation plugin for YUI 3 DataTable. This also works with pagination, so you can save all the values on all the pages in a single operation, but it is not designed to let you easily add or remove rows.

The state preservation plugin is also useful if you want to do something simpler like displaying checkboxes for selecting rows in a table.

The reason for all these different solutions is that the engineering trade-offs are quite complex, so it is not advisable to build a single kitchen sink solution to support all the above features. Hopefully, this overview will help you choose the appropriate module for tackling your next big table-based application.

About the author: John Lindal (@jafl5272 on Twitter) is one of the lead engineers constructing the foundation on which Yahoo! APT is built. Previously, he worked on the Yahoo! Publisher Network.

DataTable has been one of YUI's most heavily used and relied upon widgets for years. In 3.5.0, DataTable got a major overhaul, resulting in some small changes to the API and some big changes to the infrastructure.

In this two part article, we'll talk about these changes, what led up to the decision to revisit the infrastructure rather than focus on features, break down some of the new and experimental concepts being explored, and finish up with a look at the plan for the future of DataTable as we see it today.

So without further ado, let's address the obvious question:

Why Refactor?

The short answer is "to make DataTable easier to use and customize". The 3.4.1 infrastructure was based on a "core" Widget model, with all additional features left to plugins or subclasses. The data was held in a Y.Recordset instance and the column configuration in a Y.Columnset instance. The Widget was responsible for creating the entire table markup. That seemed like a reasonable architecture, but there were a few issues with this:

1. DataTables are defined by their features, which means there could be a lot of plug()ing to get the DataTable that fits your application. This made working with DataTable too complicated and verbose.
2. The contract of the Plugin API hindered DataTable's multiple features from interacting properly. In fact, certain DataTable features in 3.4.1 had already started breaking this contract.
3. DataTable was hard coded to use Y.Record, Y.Recordset, Y.Column, and Y.Columnset. This coupling limited the ability to customize DataTable or integrate it with other components in your application.
4. The rendering algorithm was flexible, but its hooks for customization were fixed. There was an opportunity to make rendering more customizable (and a whole lot faster) without requiring implementers to do major surgery on their instances.

The new architecture aims to address these and other issues uncovered during its production life prior to 3.5.0, and take advantage of the App Framework components that weren't around when DataTable was initially created in 3.3.0.

DataTable is incredibly important to a lot of people, and it's important to us that it not only be built right, but also that it be as easy to use as possible. These changes aim to move DataTable in the right direction.

Big Changes

The major changes from 3.4.1 to 3.5.0 are mostly under the hood, and with any luck, migrating to 3.5.0 DataTable should be painless.

Most of the outward changes to the API make it much easier to add and use DataTable features. Compare a 3.4.1 sortable DataTable to its 3.5.0 equivalent:

What's in a Name?

The first thing you'll notice is that as of 3.5.0, you create instances of Y.DataTable rather than Y.DataTable.Base. The base class is still around, but now its primary role is as a superclass for extension (which we'll talk more about in the next article).

Y.DataTable is now the main class and the namespace for classes that support DataTable, such as Y.DataTable.Base, Y.DataTable.Sortable, and Y.DataTable.BodyView. The notable exception is Y.Plugin.DataTableDataSource, which hasn't yet been migrated over to the new world order (expect that in 3.6.0).

You'll see more about this in the next article when we talk about some of DataTable's new concepts.

Features as Class Extensions

The 3.5.0 architecture marks a shift away from the plugin model for DataTable. Instead, we're putting features in class extensions, and using the powerful Y.Base.mix() method to augment these features into the DataTable class, creating a sort of ad-hoc multiple inheritance model.

The two primary benefits this approach has are:

1. The API for your DataTable and all its features is in one place, your Y.DataTable instance.
2. Rather than having to explicitly build up your DataTable from the Base, simply including the feature modules in your use() line adds support for those features to the Y.DataTable class.

But what if you have a couple of DataTables on the page, and you don't want all of them sortable or scrollable, etc.?

To keep features from leaking between DataTable instances, each feature includes a behavioral trigger, which is responsible for activating the feature for that particular instance. A feature might look for assigned table attributes, such as "scrollable", or column configurations such as "width". It might even have multiple triggers, such as datatable-sort looking for either the "sortable" table attribute or "sortable" column configuration property.

No More Coupling

As of 3.5.0, Y.Columnset and Y.Column are gone, and you're not limited to data storage in Y.Record and Y.Recordset.

Column definitions are now stored as a simple array of objects, just like what you pass into the DataTable's columns attribute.

The reason for this change is that many features are column-specific, making the column definitions a natural place to configure them. But the Column class shouldn't include feature related attributes unless that feature is requested. However, features implemented as plugins should not directly modify classes. So how can a column definition include sortable: true if the Column class doesn't have a sortable attribute and the plugin can't add it? There wasn't a good answer, so we (regrettably) added the feature attributes to the base Column class, allowing them to do nothing unless the plugin was added. With so many pending DataTable features wanting to be configured in the column definitions, we decided to remove the Column class wrappers in 3.5.0 to allow columns to be defined with whatever properties were needed. The jury is still out on this change, so we're interested in your feedback.

On the data side of things, Recordset and Record were technically working fine, but with the advent of Y.Model and Y.ModelList in 3.4.0, there was now a huge opportunity to share classes that were likely to be used elsewhere in the application. This meant, however, that DataTable would need to be decoupled from its data storage classes because the common use case for the App Framework components is to create subclasses of them with attributes and APIs that specifically encapsulate your business logic.

So in 3.5.0, rather than being bound to Y.Record, data records are stored in Model subclass instances. By default, DataTable will create a Model subclass for you, customized to your data, but you now have the option to specify the class to use by setting the DataTable's recordType attribute. Similarly, you can provide your table's data in a ModelList or ModelList-like object.

Now if you share your ModelList or Models across various parts of your application, modifications to any of them will automatically update the table.

Customizable Rendering

In 3.5.0, the rendering algorithm was completely rewritten, and is now significantly faster and more configurable than its 3.4.1 counterpart. The migration guide and user guide go into detail about the various cell formatting options, but one important change worth talking about here is that the entire algorithm can be replaced with a simple configuration change.

In keeping with the move to configurable data storage classes, DataTable now delegates its rendering algorithms to Y.View classes. In 3.5.0, the basic, featureless DataTable has attributes headerView, bodyView, and footerView. By default, footerView is unset, but headerView is set to Y.DataTable.HeaderView, and bodyView is set to Y.DataTable.BodyView.

When a DataTable instance is render()ed, the only DOM content the Widget itself is responsible for is the <table> and its immediate children. The rendering of the header content, footer content, and data rows is left to whatever View class is supplied to the respective attribute.

The default Views provided in 3.5.0 include enough column configuration options to handle most use cases, but if you have special requirements for your table rendering or you want to hyper-optimize your specific implementation, you can supply your own View classes to the attributes noted above, and DataTable will use them instead.

More to Come

The next article will go into more detail about DataTable's architectural techniques, experimental patterns, and what's coming in the next release.

In the mean time, be sure to let us know in the yuilibrary.com forum, on Twitter, and in the #yui IRC channel on freenode, how things are going in your DataTable apps and what features you're most looking forward to.

To me, it feels like every other Open Hours is about DataTable, but I guess that’s not actually true. So let’s talk about DataTable!

In particular, I want to talk about two things:

1. Tips and tricks with column configurations and cell formatters
2. What’s fast, what’s slow, and how to maximize DT performance

It will be a lot of live coding on jsfiddle, though I’ll try to have a few examples of things prepped. If there’s a particular example you’d like to see, a formatting question you’d like answered, or generally any other DT question or feedback, please bring it.

Recording

The recording is available in the YUILibrary YouTube channel, though the quality didn’t come out very well.

Mentioned

• Flyweight Proxy ModelList
• More precise DOM updates from data changes
• Custom bodyView with simpler rendering logic

A lot has been written about how abstractions simplify the job of constructing software, when programming in the large. Unfortunately, abstractions leak, as demonstrated by Joel Spolsky and Jeff Atwood. This is unsurprising, given the definition of abstraction: the process of considering something independently of its associations, attributes, or concrete accompaniments. In other words, abstractions ignore details, and this always comes back to bite us.

Generalization, on the other hand, is useful when focusing on a specific problem, i.e., programming in the small. The goal of generalization is to make a solution for one problem usable for a larger class of related problems. This is the ultimate in code reuse: build it once and use it over and over again. As long as the generalization is done correctly, there is no danger of it failing, though there is always the possibility of not taking it far enough.

There are many ways to generalize code. This article will focus on building higher order functions, a particular strength of functional languages. Let us start with two obvious examples: map and reduce which apply a function to each element in a collection. These generalize the concepts of generating a new collection or value, respectively, from an existing collection.

It should be possible to generalize map and reduce further, to work on a forest, i.e., a collection of trees, instead of only a collection of items. The problem, however, is that the code that recurses must know how the trees are stored. Is a tree simply nested arrays, e.g., [1, [2, 3], 4], or is each node an object from which the array of child nodes must be extracted? For reduce, one can pass in a function that extracts the children:

function reduceForest(roots, initial, operation, children)
{
	return Y.Array.reduce(roots, initial, function(value, root)
	{
		return reduceForest(children(root), operation(value, root), operation, children);
	});
}

For map, however, it is more complicated, because there is also the question of what the result should look like. A nested array can be mapped to another nested array, and a tree of objects can be mapped to another tree of objects, but supporting both at the same time would make the code quite complicated. Knowing when to stop generalizing is just as important as knowing when to keep going!

We can branch out in a different direction, however, because there are many types of collections, but the above code requires the children to be stored in arrays. We need to generalize the concept of iteration.

YUI’s oop.js provides this, in the private function dispatch. Here is the slightly adjusted version used by gallery-funcprog:

function dispatch(action, o)
{
	var args = Y.Array(arguments, 1, true);
	switch (Y.Array.test(o))
	{
		case 1:
			return Y.Array[action].apply(null, args);
		case 2:
			args[0] = Y.Array(o, 0, true);
			return Y.Array[action].apply(null, args);
		default:
			if (o && o[action] && o !== Y)
			{
				args.shift();
				return o[action].apply(o, args);
			}
			else
			{
				return Y.Object[action].apply(null, args);
			}
	}
}

This is wonderfully general. It works for any action: map, reduce, filter, etc. Arrays are routed to Y.Array. Objects that implement the action are called directly, allowing individual classes to optimize individual actions. All other objects are operated on by the generic versions in Y.Object (provided by gallery-object-extras). Sharp readers will note that the order of iteration for object members is undefined, but this doesn’t matter for map, reduce with commutative operators, filter, etc. (Use Y.some at your own risk, however.)

The option for objects to implement custom versions of the actions leads to yet another generalization: gallery-iterable-extras. This mixin implements all the actions for any object that provides the iterator method. The only requirement is that iterator must return an object that exposes next and atEnd. This is especially efficient for linked lists, where indexed lookup is O(n), but it could also simplify other classes, e.g., Y.NodeList, because one then does not have to explicitly implement map, reduce, filter, etc.

Of course, generalization doesn’t have to be this complicated. When I was writing gallery-sort-extras, I first built this function:

Y.Sort.compareKeyAsString = function(key, a,b)
{
    return compareAsString(a[key], b[key]);
};

But then I realized that I would have to write a separate compareKeyAsNumber, so instead I generalized it to:

Y.Sort.compareKeyAs = function(f, key, a,b)
{
    return f(a[key], b[key]);
};

Since sort requires a function that takes only (a,b), one must use Y.bind to fix the first two arguments:

sort(Y.bind(Y.Sort.compareAsKey, null, Y.Sort.compareAsString, key))

So far, we have only considered functions which operate on functions. One can also build functions that return functions. A simple example is a function to reverse the sort order:

Y.Sort.flip = function(f)
{
    return function(a,b)
    {
        return f(b,a);
    };
};

Hopefully, these examples of using functional programming to generalize code will inspire you to look for situations where you can do the same in your own code.

About the author: John Lindal (@jafl5272 on Twitter) is one of the lead engineers constructing the foundation on which Yahoo! APT is built. Previously, he worked on the Yahoo! Publisher Network.

Did you know that you can help us with this?

All of our documentation is available in our Github repo and the tools we use to generate this documentation are available for you to use. Yes, that’s right, everything we use to make our documentation is available right now. In this article I will show you how you can fix API documentation and update an example to help other developers just like you.

This article is about modifying existing examples, landing pages and API documentation. Check out the screencast at the bottom if you’re more interested in creating new examples from scratch.

What do you need?

You need a working install of Node.js (0.6.x or higher is recommended), NPM (packaged with node), Selleck our custom documentation tool and YUIDoc our api documentation tool. These tools are freely available and very easy to install. Simply go here and choose your environment to install Node.js. Once it’s installed, you can install our two tools with this command:

npm -g install selleck yuidocjs

Once that completes, you’re all set!

** You will, of course, need git installed, have a Github account and have already forked the yui3 repo

Where things live

All of our examples are kept in our main source tree, this makes it easier to associate an example with the code it belongs to. In this example, I will be working with the DragDrop examples and API docs.

The main landing page and example files are located under yui3/src/dd/docs. All of the API docs are parsed from the raw source files (we’ll get into that in a bit).

Seeing a rendered example

The hardest part of any example is seeing how it looks once it’s ready. This is where Selleck comes in to play. Selleck has a “server mode” that you can fire up and see our examples “parsed and loaded”. Turning on Selleck’s“server mode” is very simple:

cd yui3/src;
selleck --server

This will print the following to the console:

[info] Selleck is now giving Ferrari rides at http://localhost:3000

Now visit http://127.0.0.1:3000 in your browser of choice and you should see the main Selleck page displaying a list of all the components that it found under the src directory.

If you don’t want Selleck to bind to port 3000, simply add a port to the command above (selleck --server 5000)

One of the advantages of using Selleck in server mode is that you do not need to restart the server (unless you add new json files for new examples) to see your changes. Just open a file, edit, save and reload! It’s that easy!

Fixing API Documentation

All of our API documentation is parsed directly from our source files yui3/src/dd/js with
YUIDoc. This makes reading them in the source files a little difficult (unless you can parse JSDoc tags and Markdown syntax in your head). Luckily YUIDoc also has a server mode to help you with this. Turning on YUIDoc’s server mode is just as easy as Selleck’s:

cd yui3/src;
yuidoc --server

This will print out some YUIDoc debugging info ending with:

info: (server): Starting server: http://127.0.0.1:3000

Now visit http://127.0.0.1:3000 in your browser of choice and you should see the main YUIDoc page listing all of the parsed API docs.

If you don’t want YUIDoc to bind to port 3000, simply add a port to the command above (yuidoc --server 5000)

YUIDoc’s server mode works a lot like Selleck’s in that you do not need to restart the server to see your changes. YUIDoc will reparse all of the source code on each page load and show you the updated API docs. Just open a file, edit, save and reload! It’s that easy!

Things to remember

Some things to remember when you are working on something you want to contribute:

• Always work in a branch git checkout -b mydocpatch
• Push to your branch git push origin mydocpatch
• Submit the Pull Request from your branch
• (optional) If you’re comfortable with git, we recommend working against yui/yui3′s “live-docs” branch

What to do after I update things?

As with anything else in YUI, once you update your files, simply commit them and issue us a Pull Request as usual. One of our developers will verify the changes and either merge them in or give you some feedback.

How often are things updated on the site?

Our current site deployment is very easy, we often deploy to yuilibrary.com at least once a week. Sometimes we even push daily! So, get your changes in and give back to the community!

More Information

YUI Engineer Luke Smith put together the following screen cast showing how to create a new example from scratch. Take a look at it and more videos over on our YouTube Channel.