Skip Navigation Links.
MISSING BANNER
Project Home >> Current Version Notes

SobekCM Current Version Notes

 

These are notes for version 4.11.0 which will be released by January 20th, 2018.  Stay tuned for the upgrade instructions. 

Download the newest version from our download center here.

Release notes on previous versions of SobekCM can be found here.

Summary

Version 4.11.0 contains a great many fixes but the largest is a new solr/lucene index and searching system.  All searches (and associated faceting) now occurs in solr, rather than metadata searches occurring within the database.  This greatly increases the speed of searching and also allows for better support of non-roman characters in the metadata and full text.

With the release of version 5.0 in mi-2018, this new searching system will completely replace the old legacy searching system, but for now, the two can operate side-by-side.  

In addition, a great number of tweaks were applied to the metadata templates and metadata display in the citation.  Fixes for bookshelves and several other features help round out this new version.

The new version is recommended for ALL instances, not just those that are ready to upgrade to the new searching system.

Solr/Lucene Index Updates

New Solr/Lucene Indexes

  • Searching metadata, full text, and date range searching all can use the new solr/lucene v5 indexes
  • Text searching now works in concert with an advanced search and shows facets and refines searches correctly.
  • If full text is included in the search, then the search results are not grouped and the highlighted text snippet appears
  • New solr monitoring software is also included in new version, to make JETTY based solr work as a Windows service
  • Improved identifier and accession number searching
  • Great many more display fields, to display the data in a more verbose or detailed manner than the indexing 
  • Lays ground-work for new search results viewers and new architecture for those viewers (to be included in next version)
  • Changed the solr indexing to work just like the database and be extensible with metadata plug-ins, etc..
  • Results Viewer changes based on new system
  • All searches, from top-level ALL browses to full text searches, not include facets ( facet and search statistics continue to be cached)
  • Unicode search terms display correctly now in the search explanation on the search result screens
  • Database changes
    • Added new flag in the settings table for the search system ( legacy or beta/v5 )
    • Added new fields for the legacy solr index urls, so we can switch between legacy and beta/v5
    • Searching in new system now fully uses the facet and results fields listed in the database tables
    • Updated metadata types in the database to include the legacy solr term, as well as the new solr search, display, and facet term for each metadata type

 

Aggregation Management Changes    

  • Added a flag at the aggregation level that will control if results are grouped by title/bibid when possible during searching
  • New item flag on the aggregations is now based on the MadePublicDate field
  • Fields to include in the search results is now stored in a new table in the database at the aggregation level and no longer uses the SQL snippet in the Item_Aggregation table
  • Facets to display at the aggregation level moved from the aggregation configuration into the database and additional override options provided
  • Adding a new aggregation copies all the facets, result views, and result fields from the parent collection

 

Search Index Metadata Changes

  • Each field in solr now supports the indexed field, the facet field (usually just the facet version of the indexed field), and a display field for showing in the browse/results.
  • Several core existing fields had new display versions added
    • 'Genre Display' added which adds the authority name to the genre for display purposes
    • 'Identifier Display' added which includes the identifier type to the identifier for display purposes, where you may not want that to display in the facets
    • 'Creator' no longer includes the role type in the indexed and faceted name
    • 'Creator.Display' added which includes the role and potentially more creator information that you would not want to appear in the facets
    • 'Name as Subject Display'  holds a display version of the name as subject, which can have more data than you would really want to index or display in the facets
    • 'Title as Subject Display'  holds a display version of the title as subject, which can have more data than you would really want to index or display in the facets
    • Added VRA Core-based Measurements Display, which can hold more information like units or potentially what is measured 
    • Added VRA Core-based Materials Display, which can hold more information like which part of the artifact if being described
  • Changes to the TITLE fields, to allow more specific searching and display
    • 'Title' metadata now includes the sub-title as well ( for both search and display purposes )
    • 'Title' field is really just the main title, but there is an 'all titles' field that can be searched as well
    • 'Other Title' added to hold all non-main titles
    • 'Translated Title' added to hold the main title translated, if present in the metadata
    • 'Series Title' holds the series title now
  • None of the learning object metadata was being indexed before.  Added the following fields to the search and display fields:
    • LOM Aggregation enumeration ( level 1 - level 4 )
    • LOM Context and LOM Context Display
    • LOM Difficulty enumeration
    • LOM Intended End User
    • LOM Intended End User Display ( lists them all in one line, like 'Author, Learner, Manager' )
    • LOM Interactivity Level enumeration 
    • LOM Interactivity Type enumeration 
    • LOM Status 
    • LOM Requirement and LOM Requirement Display 
    • LOM Age Range 
    • LOM Resource Type and LOM Resource Type Display 
    • LOM Requirement Display 
    • LOM Learning Time
  • The performance art specific metadata was not being indexed before either.  Added the following to the search and display fields:
    • Performer
    • Performance
    • Performance Date
  • Added zoological ( Darwin Core-based ) hierarchical field which includes the entire hierarchy for display purposes mostly
  • Added another serial hierarchy field which will be used to dynamically pull the next levels in the hierarchy tree as facets that combines the index and text ( i.e., '00001|January' for example )
  • Date changes
    • Refined and corrected the way publication date is indexed (with display date and gregorian date and date year ) which will allow other calendars to be used for display, and then mapped into Gregorian for sorting, etc..
    • Added new timeline date solr fields, which will use the temporal subject or the publication date for indexing purposes (into a single authoritative date field)
    • Added more refined temporal subject solr fields
  • Other new administrative fields in the solr schema include:
    • made_public_date - indicates the date the item was first made public 
    • hidden - flag indicates the item is hidden from discovery entirely
    • discover_users - will include the UserIDs for all users who have rights to discover this item ( or -1 if no user-specific restrictions )
    • discover_groups - will include the GroupIDs for all users groups that have rights to discover this item ( or -1 if no group-specific restrictions apply )
    • discover_ips - will include the primary key to the IP groups that can discover this item ( or -1 if no IP-specific restrictions apply )
    • restricted_msg - will allow the restricted message to be customized for each item 
  • Several fields were added to support the authority system planned for this upcoming year
    • creator.authid - internal authority id that can include links to one or more external authority systems for named entities
    • publisher.authid - internal authority id that can include links to one or more external authority systems for publishers
    • subject.authid - internal authority id that can include links to one or more external authority systems for subjects
    • spatial.authid - internal authority id that can include links to one or more external authority systems for spatial entities
    • temporal.authid - internal authority id that can include links to external authority system(s) for temporal entities

 

Other Changes and Updates

Builder changes

  • Minor change better catches and reports a top-level exception in the builder when rebuilding all the static aggregation browses ( in the RebuildAllAggregationBrowsesModule )
  • Tweaked support to handle large JPEGs
  • Corrected an issue preventing the builder from correctly reloading the config if it changed while the builder was running.  Change to worker controller to check for latest config at the beginning of each polling and then repull the info and basically start over.  Update fixes an issue in the builder that called for a full reload of the configuration all the time.. not just when the configuration had changed.
  • Corrected the MoveFilesToImageServerModule in the builder to not move the METS file if it is just a reprocess, but to make a backup as well
  • Builder now correctly deprecates builder modules that are not enabled
  • We now have TWO solr/lucene builder modules.. one for legacy and one for the new indexes

 

Updates related to the online metadata templates 

  • More support for defaults in the template configuration files
    • Combo box / Text box elements now apply defaults when they are present in the templates
    • Made the default values for text area elements protected, so classes that extend it can access the defaults
    • Added the ability to apply a constant value to the following elements
      • ETD Degree Grantor
      • ETD Degree Level
      • Holding Location
      • Rights Element
    • Made the aggregation list in the source institution case insensitive, which means it works for defaults correctly now
    • Made the resource type element apply a default from the template XML correctly
  • Added the ability to support non-standard resource types, as a sobekcm genre ( such as thesis )
  • Added new date fields and template elements for MODS origin_info dates ( SOAS-53 )
    • Date created ( field existed, just added template )
    • Date modified
    • Date valid
    • Date captured
  • Added new template element for accession number (separate from identifier ) and added to edit templates
  • Graduation semester is now correctly saved in to the METS files
  • Code Fixes
    • Empty VRA core measurements and materials are no longer added into the object, which resulted in an empty VRA core METS section at times  
    • Corrected an issue loading template elements in the base code but not referenced in the Element_Factory class
    • Corrected some issues with the new template date elements and put in the Element_Factory to avoid reflection when possible
  • Saving changes now correctly updates the index and builder
    • When an item is saved now with metadata changes, the builder is alerted correctly   
    • As soon as items are entered or edited, the solr configuration is updated

 

Search Result changes

  • Corrected an issue that selecting a facet did not reset the page in the search results displayed ( SOAS-46 )
  • fix to make results caching based on results per page. Search results are cached in a way that the number of results per page is important
  • Reduced the caching on the results data, so changes reflect more quickly down to 1 minute ( SOAS-72 )
  • Disabled the export view (temporarily) in the configuration   
  • If no results viewer can be found, from the URL viewer code, in a search results display, the BRIEF display is used. This stops an exception for a null resultsviewer which was sending emails to the admins.

 

Item Display changes

  • Citation
    • Added a custom citation section writer for the list of aggregations linked to an item and corrects link ( SOAS-71 )
    • Graduation date and graduation semester were added to the citation configuration to display when present
    • Graduation date and graduation semester were added to the BriefItem mapper for the ETD data as well
    • Materials (from VRA core) now maps to display in the citation with the 'materials described' displaying
    • Measurements (from VRA core ) now maps to display in the citation with the separate 'units' displaying
  • Corrected the TEI plug-in configuration ( ItemWriter tag replaced with ItemWriterConfig tag ) 
  • Corrected an issue that switching between JPEG and JPEG2000 from the menu reset the current page ( SOAS-44 )
  • Corrected an issue displaying PDFs in HTTPS mode ( SOAS-74 )   
  • Updated SQL to make the serial hierarchy support non-roman characters ( SOAS-11 )
  • Snippets for searching within a document are now much longer and work correctly again

 

Statistics

  • Added a fix to stop web content usage from having all pages appear in stats and fix old stats ( SOAS-82 )
  • Fixed an issue with the item count when a date ( or FYTD ) was included.. item count was wrong in option 2
  • Fixed caching issue that made the resource count stats not report correctly when the option was changed

 

Aggregation Management and Display

  • Aggregation list now allows right-click and open in new window on code and name, since they are links ( SOAS-57 )
  • Private items listing now has a checkbox which opens the item in a new window ( SOAS-18 )
  • Aggregations now correctly show the unicode name for child collections in the buttons (SOBEK-381)
  • Added ability to set the full text checkbox as default for a collection by setting the 'Basic_Text_Search_Combined_AggregationViewer.Include_Text_By_Default' setting to 'true' for that collection     

 

Changes to Support Importing Data by Spreadsheet (very early phases) 

  • Added a new mySobek viewer to get all the data import information ( i.e., filename, upload, mappings, constants )
  • Added ClosedXML and necessary support DLLs to read the Excel files
  • If you upload an Excel file, you can then select the worksheet to import
  • Will next be working on selecting the column mapping (for Excel and CSV) and adding constants (for Excel, CSV, and Marc)
  • Added new fields into the solr indexing to support complex, hierarchical entities mapping into multiple, related columns when exporting metadata back into a spreadsheet from the system.  These support better round-trip ( i.e., export metadata into spreadsheet, update, and then re-import the spreadsheet into the system ) without losing sub-elements.
    • creator_export ( includes name | role | affiliation | authid )
    • subjects_export 
    • spatial_export
    • identifier_export ( includes identifier | type )
    • genre_export ( includes term | authority )
    • notes_export ( includes notes | language | type )
    • title_export ( includes title | language | type )
    • classification_export ( includes term | authority )
    • temporal_export

 

Other

  • Bookshelf corrections
    • Corrected issue with the submitted items / bookshelves display ( SOAS-79, SOBEK-375 )
    • Corrected several bookshelf behaviors
    • Bookshelf was not launching from menu correctly
    • Bookshelf in thumbnail view was not displaying properly, due to missing javascript for thumbnails
  • Update corrects if the usage email source file is missing under the design folder.. settings no longer throws a 404 ( SOAS-79 ) 
  • Correction to the downloads item viewer, which makes the SEO robot output more valid (had an added closing tag before)
  • Attempting to get the sort safe date was throwing an uncaught exception if the date looked valid, but was not actually
  • Minor updates to several CSS files ( sobekcm.css, sobekcm_item.css, and sobekcm_metadata.css ) 
  • Added some code to make the system pull 1000 results per page for the timeline results viewer
  • Updated the web and builder to .NET framework 4.5.2, from 4.5
  • For errors emailed or stored in the exceptions log, the URL referrer is attempted to be added