Tuesday, August 29, 2017

Taxonomies in SharePoint

Controlled vocabulary metadata, including hierarchical taxonomies, has been supported in SharePoint since its 2010 version, and its use and features have been enhanced is succeeding versions of SharePoint. While it’s not technically difficult for users to create taxonomies and apply their terms to content items in SharePoint, developing a metadata/taxonomy design and application strategy is definitely a challenge.

The distinction and overlap between metadata and taxonomies was the topic of my previous blog post, "Metadata and Taxonomies," and it is very relevant to SharePoint. Controlled vocabularies or taxonomies used to tag content are referred to in SharePoint as “managed metadata.” This designation is indeed accurate and fitting. Some, but not all, metadata is in taxonomy form (hierarchical structures), and in SharePoint it is managed/controlled in a central way, where permissions on who can change or add to the metadata terms may be limited to a smaller set of users than those who may tag content with the metadata. “Managed Metadata” is something you will hear about in documentation, but in the SharePoint application itself, what you want to work with is “Term store management,” grouped with other Site Administration settings under “Site Settings” (under the gear symbol in Office 365 SharePoint). Terms are grouped into “Term Sets” (top-term hierarchies or facets).
Questions to consider in taxonomy design in SharePoint include:

  • To what extent will document libraries (virtual folders) be used to categorize content within a site, and would proposed subfolder names be better suited as metadata terms for tagging?
  • Will the primary use be for filtering lists of documents in place, within an open document library, based on metadata selected for the various “columns,” or will the primary use be for refining search results, based on metadata selected in the left-hand margin refinement panel after executing a search?
  • How many Term Sets should be created and how many and which metadata fields in total should display to the users, either in columns or as search refinements.
  • When should a Term Set be a flat list and when should it be created as a hierarchy, and how deep should the hierarchy be?

Use of document libraries vs. metadata tagging 


SharePoint supports the creation of a hierarchy of nested folders within libraries within sites. So, it may be tempting to start of creating such a “taxonomy” of categories for content, especially if migrating content over from a shared drive where such folders had been used. However, tagged metadata has many advantages over categories of folders for finding and retrieving content.
A content item may be tagged with more than one term from the same Term Set if it deals with more than one topic or if it falls into more than one category type, whereas putting a document in more than one folder can lead to version control issues. (It’s true that you can put a document in one folder and a link to it from within another folder, but this is not easily remembered to do, nor does it look as “clean.”)

You can create Term Sets (as facets) each for multiple ways to categorize, such as by document type, function, audience, topic, etc., serving as facets, and then tag a content item with terms from each, whereas folders don’t deal well with mixed methods of categorization, and you are forced to choose one method of categorization.

  • Tagged metadata allows you to filter a large set of content in place to quickly narrow results to what you want, whereas folders require clicking down through multiple paths, taking more time to find desired content, which is in different places.
  • Tagged metadata can also be implemented as search refinement filters, also known as faceted search.
  • Tagged metadata terms can have synonyms, helping users find what they want by different names, whereas folder names cannot have synonyms. 

Thus, what had been labels for folders on a shared drive should most likely be changed to terms in taxonomy Term Set. Whether you should have any document libraries, or just a few without subfolders, depends on the preferences of your users, but I don’t recommend the creation of subfolders. 

Filtering on columns vs. refining searches


The same Term Sets may be used for both column filters and search refinements. But typically, the implementation of managed metadata in SharePoint is either primarily for one or the other purpose, and the other use may not even be set up. Generally, if the managed metadata is going to be applied to documents within a single library or one site, on the order of tens or hundreds, then column filters are desired; if the managed metadata is going to be applied across multiple sites on thousands of documents, then search refinements would be used.

If unsure whether to promote filtering on columns or refinements on search, consider that filtering columns will always get more accurate results, but metadata has to be consistently applied.  Out-of-the-box search in SharePoint will retrieve documents with the word or phrase anywhere in the document. The idea behind this is to get search set that is larger than needed and not miss anything, and then the user can refine the search result with the various refinements. So, the results of search are not as accurate, but there will be results, even if metadata tagging is incomplete.

Knowing how the Term Sets will be used can have an impact on the wording of terms and the extent of use of hierarchy. Both columns and refiners have limited width for term names to display. The user can easily adjust column widths to accommodate long names, but the refinement panel width cannot be widened by the user. The use of columns also makes it desirable to keep terms to a limited length within a given Term Set.  Refiners indicate hierarchies of terms by default to the user who is searching content, whereas columns do not indicate any hierarchy in the default view.

Number of terms sets and metadata fields


There is no point in creating a Term Set if it’s not going to be displayed to the users for filtering or refining, and too many metadata fields take up horizontal or vertical space, are a burden to tag, and make the user experience of searching or filtering too complicated. So, you need to consider what would be truly useful, and not merely possibly nice to have. Just two Term Sets, such as Document Type and Topic, may be sufficient. In addition to the managed metadata that you create, there will be other metadata fields desired for filtering or refining, such as date, author, and format type, and perhaps uncontrolled keyword tags applied by users. In the case of columns, there will always be the document title taking up a column and considerable horizontal space as well. 

The default columns in SharePoint are “Type” (file format) “Name” (filename), “Modified” (the date the file was uploaded or the last time any of its properties were updated, not the date the file itself was modified) and “Modified By” (the person who uploaded the file or last updated its properties, but not necessarily who actually modified the file). The default search refinements are the same, excluding title: “Result type” (file format), “Author” (an even worse misnomer for Modified by”), and “Modified date” (often displayed in a graph form). If you believe such information is not that valuable, you can remove these columns/refinements, especially when you plan to add other columns/refinements, which will take up horizontal or vertical space. 

I would recommend no more than 4-7 total metadata fields, including those that are not based on managed metadata. You should avoid having more metadata as columns, along with the document titles, than can fully display horizontally, so as not to require horizontal scrolling. Search refinements, on the other hand, by default display sample high-use terms under each refiner, so typically no more than three refiners display in the left margin without vertical scrolling. Vertical scrolling is expected and acceptable to a limited degree.

Term Sets as flat lists or hierarchical taxonomies


SharePoint makes it easy to create hierarchies within Term Sets by simply right-clicking on a selected term and selecting “Create Term” from the context menu. Some people might thing that since the Term Store is for taxonomies, and taxonomies are hierarchical, hierarchies should be created if applicable. However, hierarchies are only helpful for navigating the taxonomy if the taxonomy is sufficiently large. If you set up multiple Term Sets, each used as a facet in combination with others, they each don’t need to be very large. Furthermore, the types of content most people store in SharePoint tends not to need extensively large and detailed taxonomies as might be needed in a content management system or digital asset management system

My rule of thumb is up to 12 terms should be on the same level before considering creating any hierarchy, but it could go up to 20 or so, and even more if the list of named entities/proper nouns.  Also, if you do have hierarchies, consider keeping them relatively shallow, such as to only two levels, instead of three. Even if a hierarchy is technically correct, it does not mean you have to set it up that way.

If you need only a short flat list of terms, you might consider not using the Term Store at all, but rather create the list as "Choice" type of column. This is easier to implement, but the terms would limited in their use to filtering and sorting the column, and could not also be applied to search and navigation. 

Monday, July 17, 2017

Metadata and Taxonomies

Metadata and taxonomies are related. In The Accidental Taxonomist, 2nd edition (pp. 15-18), I explain that most, but not all, taxonomies (not purely navigational taxonomies) serve to populate terms/values in metadata fields/elements; and some, but definitely not all, metadata fields are populated by terms/values from controlled vocabularies or, more specifically, taxonomies (in contrast to free text or key words).

The question remains whether to start with creating the overall metadata strategy and schema and then build taxonomies as part of it as needed, or to start with creating a taxonomy and then, in the process, identify the various descriptive metadata.  Ideally the two are developed for implementation combination, as part of an integrated strategy. However, an expert in taxonomy development (a taxonomist) and an expert in metadata design (a metadata architect) are usually not the same person.

A metadata architect can become an accidental taxonomist, and a taxonomist can become an accidental metadata architect, or the two experts can work together on the same project, although it is not so common for an organization to have both such experts on staff.  Whether an organization has a metadata architect or taxonomist depends on the nature of the organization’s content and content organization needs.

Organizations that start with the metadata expertise and approach to information management tend to be those with significant needs in digital asset management (with image or other media collections), records management (in highly regulated industries), publishing, or cultural preservation (museums or libraries). Organizations that start with the taxonomy expertise and approach include product or service providers, distributors and retailers (especially in ecommerce), and organizations focused on providing information resources.

A hierarchical taxonomy can be integrated with metadata, when one of the metadata fields is for “Topic” or “Subject,” and there is a hierarchical taxonomy of subject terms associated with that field. However, it is the faceted type of taxonomy in particular that unites the tasks of taxonomy creation and metadata design.

Faceted Taxonomies and Metadata

A faceted taxonomy comprises a set of facets, each an individual controlled vocabulary, whose terms are generally not linked/related to terms in the other controlled vocabulary facets, but the combination of terms from a combination of facets are used to tag the same set of content, and users search/filter on terms in combination from various facets. Examples of facets may be Product/Service, Market Segment, Location, Document Type, Supplier, etc. A faceted taxonomy is a common type for both enterprise taxonomies and ecommerce or product review taxonomies, and it’s a type of taxonomy that taxonomists are familiar with creating. It’s called a “taxonomy” even though it differs from the classical hierarchical “tree” type of taxonomy, because it involves controlled vocabulary and classification. The name for each facet and the terms within the facet constitutes a simply two-level hierarchy.

Each facet is also a metadata field/element. The taxonomist designing a faceted taxonomy is thus also designing metadata, at least some of it. There are usually more metadata fields to describe the content beyond those which comprise the taxonomy facets. For a faceted taxonomy to best serve the user who is trying to find/discover content based on what it is and what it is about, the number of facets should be limited. (See my earlier post "How Many Facets.") Metadata, however, can serve additional purposes beyond helping users find content. Metadata may describe content for purposes of full identification, source citation, or information on how the content can be used, including rights data.  The taxonomist or metadata architect needs to decide which metadata fields will constitute a displayed faceted taxonomy for the end-user to utilize in search/discovery, and which metadata fields will not but will rather display on a selected content record.

On the other hand, there may be additional metadata fields beyond the scope and definition of “taxonomy” that are nevertheless made available to the end-user to filter/refine results alongside the other, taxonomy facets. These could be for author/creator, date, title keyword, text keyword, file format, etc. Sometimes the distinction between taxonomy facet and other metadata in this case is not so clear, such as for Document/Content Type, Audience, or Language, when these fields utilize controlled vocabularies. Due to this overlap and blurred distinction between taxonomy facets and displayed metadata for filtering, it is a good idea to design the taxonomy and metadata together as an integrated strategy.

Sunday, June 18, 2017

Standards for Taxonomies

Since “taxonomies” are rather loosely defined, standards specifically for taxonomies do not exist, but there are standards that are relevant to taxonomies. A taxonomy is a kind of controlled vocabulary, and there are standards for controlled vocabularies. There are also standards specifically for thesauri, a kind of controlled vocabulary with which taxonomies typically share many features. 

Standards serve various purposes. Two leading purposes for standards are:
  1. To ensure consistency and ease of use across different products or systems used by different users.
  2. To ensure interoperability, the sharing or exchange of products/services/information.

Standards for Consistency

Standards aimed at ensuring consistency and ease of use would include buttons on devices, menus in user interfaces, pedals in cars. With such standards, users can expect the same experience from manufacturers or service providers and thus they are able to easily use products or systems from different manufacturers/providers/vendors. In the case of information systems, this kind of standard includes those for the design and style of book indexes and thesauri. These “standards” tend to be guidelines, recommendations, or accepted conventions, and not exactly strict standards, even if issued from a standards body. For thesauri, the “standard” is issued by the National Information Standards Institute (NISO), but it is called a "guideline”: ANSI/NISOZ.39.19 Guidelines for the Construction of Monolingual Controlled Vocabularies. The corresponding ISO standard is ISO 25964 Part 1: Thesauri for Information Retrieval.

These guidelines cover style and form of terms, circumstances for creating the various kinds of relationships between terms, use of notes on terms, etc. They are all about how to create well-formed thesauri with consistent design features that are then easy and intuitive to use. For example, when a user sees that two terms are in a hierarchical relationship, the user understands that the narrower term is a kind of, instance of, or integral part of the broader term, and not merely an aspect of or some other related concept of the broader term. In fact, the end-user of a thesaurus does not even need to know and understand thesaurus principles to be able to make use of a thesaurus to find desired concepts and content.

Standards for Interoperability

The other kind of standards, those aimed at ensuring interoperability, would include standards for size and units of measure, data exchange, and communications protocols. Interoperability standards are important for those controlled vocabularies which are intended to be shared or reused. Thus, the content to which controlled vocabularies link can be accessed by third parties or made publicly accessible over the Web. Controlled vocabularies may be “reused”, if the original creator of a controlled vocabulary decides to license the vocabulary (without linked content) to other publishers to use on their own content, so that the second publisher does not have to reinvent a controlled vocabulary that already exists in same subject area.

Interoperability standards for controlled vocabularies include ZThes (a thesaurus schema for XML, which is has since gone out of style), World Wide Web Consortium (W3C) specifications for the Semantic Web including SKOS (Simple KnowledgeOrganization System) and the Web Ontology Language (OWL) for ontologies, and ISO 25964 Part 2: Interoperability with other vocabularies. Indeed, ISO 25964 covers consistency standards in its first part and interoperability standards in its second part. 

Metadata Schema

Since taxonomies or other controlled vocabularies may be used to provide terms that fill a certain metadata element/property/field within a larger set of metadata, the use of a standard metadata schema or model is yet another way in which interoperable standards involve taxonomies.  If structured content is to be shared or exchanged, the metadata fields need to be standardized with the same names, abbreviations, and purposes.

Examples of standard metadata schema include MARC for library materials, Dublin Core (DCMI) for generic online networked resources, IPTC (International Press Telecommunications Council) for photographs and other media, DDI (Data Documentation Initiative) for describing data from the social sciences, and PREMIS (Preservation Metadata: Implementation Strategies) for repositories of digital objects. Adopting such a metadata schema would be another way to enable sharing of content tagged with the metadata.

I was pleased to have the opportunity to learn more about information and publishing standards recently at the Society for Scholarly Publishing conference in attending pre-meeting seminar “All About Standards.”

Wednesday, May 24, 2017

Adjective and Verb Terms in Taxonomies

Terms in a taxonomy are generally nouns or noun phrases, but this does not mean that a taxonomy cannot comprise adjectives or verbs instead. There may be differences of opinion on this, though.

A thesaurus, another kind of controlled vocabulary, by contrast, is expected to follow standards (ANSI/NISO Z.39.19 or ISO 25964), which dictate that the terms be only nouns or noun phrases. Since a thesaurus is more structured than a taxonomy, it might be assumed that a thesaurus is a kind of taxonomy with additional features (nonpreferred terms, associative relationships, scope notes, etc.), but that the basic format of the terms are the same.  In general, this is true. Terms in the vast majority of taxonomies follow the same format as terms in thesauri, and the differences between these two different knowledge organization systems lie rather in their use of term relationships and additional attributes.

Taxonomists should attempt to follow the thesaurus standards when creating taxonomies, to the extent that is practical or relevant. Reflecting the content and serving the users are always the first priorities for taxonomies. So, there may be cases when terms as adjectives or verbs are practical.

Taxonomies vary more than thesauri do, though. While the structure of a thesaurus is consistent, taxonomies can be based on hierarchies or on facets or a combination of both. Facets are lists of terms to describe certain attributes, aspects, limit-by/filter-by categories, or metadata fields. Facets could include types such as color, size, speed, etc., in which the terms in these facets are adjectives, for example the names of individual colors.

Taxonomies with terms that are verbs are even less common than taxonomies with terms that are adjectives. Taxonomy terms of verbs (not merely verbal nouns ending in -ing) are found in only very special-purpose taxonomies. As with taxonomies with adjectives, the verb terms would not comprise or be scattered throughout an entire hierarchical taxonomy, but would rather serve as shorter term lists or facets. A good example, is Bloom’s taxonomy of educational outcomes, which is just the short list of the following verbs in this order: Remember, Understand, Apply, Analyze, Evaluate, and Create. Taxonomists might dismiss Bloom’s as not really a “taxonomy,” but it is very common to use Bloom’s terms in a facet within a faceted taxonomy for educational content.

Sets of longer verb phrases may stretch the definition of taxonomy or controlled vocabulary, but they still serve the same purpose of a controlled list within a metadata property used to tag content. This is the case for learning objectives used to tag educational content. An example of a learning objective is: “Classify costs as direct versus indirect.” Learning objectives can even be put into a hierarchy, like other taxonomies.

Metadata of phrases that begin with verbs could also be used to describe processes or procedures. I had been asked once to design a “taxonomy” for the steps and options of statements/questions to be made by sales representatives as they go through the process of achieving a sale. These “terms” would have been verbal statements similarly complex as learning objectives. The issue I had with calling it a taxonomy is that the statements would not be arranged hierarchically of broader/narrower, but rather in a flow-chart procedure format. Indeed, this would have violated the definition of a taxonomy which has to have some hierarchy. However, this would have resemblance to an ontology with its semantic relationships. So, such a procedure system still would be a kind of knowledge organization system.

Sunday, April 23, 2017

Taxonomy Term Specificity

One of the challenges in creating or editing taxonomies is determining how specific the terms should be. This is a key issue in making a taxonomy customized for a certain implementation, which involves a unique set of content to be tagged/indexed and a certain set of users. Highly specific terms tend to be the consequence of deeper hierarchies. So, the decision of how specific the terms should be is also related to the decision of how many hierarchical levels of depth the taxonomy should be. Taxonomies that are organized into multiple facets, on the other hand, tend to have more limited hierarchy, if any, and terms that are not so specific.

Having taxonomy terms that are more specific than necessary inevitably means that there are more taxonomy terms than necessary. The larger taxonomy is more difficult to maintain both in currency and consistency. Terms that are more specific than necessary are also likely to be more specific than expected by the users and might get overlooked and not even used. If the taxonomy is searched, the users will not likely search for such highly specific terms. If the taxonomy is browsed, the users might stop at a higher-level broader term and be satisfied with that. Furthermore, users like to retrieve multiple results (content items or references) for a single search term, so that they can browse the list and evaluate what they want. Highly specific terms will match fewer content items, so retrieved results could comprise only one or two items per taxonomy term, which may not satisfy most users. Having a greater number of more specific terms can also lead to more inconsistency in the indexing/tagging, whether manual or automated. 

Having taxonomy terms that are not specific enough means that each taxonomy term is indexed to a relatively large number of content items, and the users may have to scroll through multiple screens of returned results and look at multiple items to find what they really want. The availability of additional filters or facets can help limit the results, though. Having terms that are not specific enough also makes it more difficult for users to “discover” potential related topics of interest, whether the terms have “related-term”/”see also” relationships between them or whether “related” terms are suggested by shared tagged occurrence among content items.

Taxonomists sometimes refer to term specificity as “granularity” or a taxonomy being “granular.” There is the irony that, although the scope and meaning of specific terms is granular/narrow/small, the terms themselves are not small. The “granular” terms tend to be longer, more complex, multi-word terms. If combining multiple concepts into a single term, such terms might also be called "pre-coordinated" terms. Following are examples of specific, granular taxonomy terms from different specialized taxonomies:

  • Possessed object access systems (in an information technology taxonomy)
  • Fingerstick blood sugar testing (in a health care taxonomy)
  • Standard manufacturing overhead cost (in a business taxonomy)

The taxonomist typically creates specific/granular terms, based on the concepts of sample content to be tagged. There may be a document with the phrase in the title, an image with the phrase in its caption, a product with this description as its type, a department with the phrase in its name, etc. Obviously, source phrases would need to be edited to become well-formed taxonomy terms, but they may still be multi-word, complex terms. Creating a taxonomy from scratch usually involves a combination of a top-down and bottom-up approach in the development of terms and hierarchical relationships. The specific/granular terms are the result of the bottom-up component of taxonomy development.

Taxonomies available for license might be appropriate in their subject area and scope, but chances are that their terms get either too specific or not specific enough for different implementations. Thus, if you choose to license a taxonomy, make sure your license allows you to customize the taxonomy so that you can either delete terms that are too specific or add more terms, as narrower terms to existing terms, that are more specific to suit your content

Creating or deleting specific terms is also part of periodic taxonomy maintenance. If a term, which has no narrower terms, is heavily used in indexing, it might be time to “break it up” be creating a few more specific, narrower terms so that the large content set is indexed and retrieved with more specific terms for more manageable result numbers. If, over a period of time, a specific terms has been applied in indexing very few times, or not at all, it should probably be deleted. The deleted term can be changed to a variant/nonpreferred term/alternative label for an existing broader concept. The specificity of a taxonomy should match the specificity of the content being tagged with it, and this can change over time.