Version 5.0 brings a number of changes users will notice directly: an integrated archival system with reporting and checksum verification, single sign-on support (OpenID and SAML), IIIF manifest and search support for sharing resources with other institutions, the ability to restrict items to specific user groups, expanded OER capabilities including new tools for authoring and viewing open textbooks, and continued UI localization with full French and Spanish translations plus new support for German, Italian, and Dutch.
In addition, performance "cold serving" digital resource is much improved. The first time a digital resource is requested, the resource is not present in the memory cache, so it would have to be built from the full METS, augmented with behavior data from the database, and converted to a transfer object. Now the final digital resource transfer object is cached. This results in first hits against a digital resource being 10-15x faster.
Just as significant, though less visible, is the work done to modernize the platform's foundation and make it easier to support and extend going forward. The application has been migrated from .NET Framework 4.7.2 to .NET 10, moving the hosting model to ASP.NET Core through Kestrel and opening the door to running SobekCM on Linux in addition to Windows. Searching now goes entirely through Solr, which allowed the metadata tables and much of the related stored-procedure logic to be removed from the database, simplifying the schema and making it easier to support PostgreSQL and RDS alongside MS SQL. Digital resource files can now be stored in GCS cloud buckets rather than only local storage, and a large amount of general tech debt was addressed along the way, including security patching, OWASP remediation, and a substantial cleanup of dead code and unused stored procedures. Together, these changes are intended to make future development, hosting, and long-term maintenance of SobekCM considerably easier.
Performance Improvements
The largest performance improvement is for the first hit against a digital resource. If an item has been retrieved in the last hour or so, the information about the item would be in the memory cache, and was very quick to retrieve and render.
However, if this is the first request for the item in a while, the item had to be built by reading the complete METS file, adding behavioral information (such as aggregations, wordmarks, viewers, etc..) from the database, and then mapped into the
leaner transfer object to be consumed by the web application. Now, that transfer object is cached in a cache.protobuf file within the digital resource folder. If the item isn't in the active memory of the
web application, then the cache.protobuf file is read rather than doing the full build.
This change increases the speed of the first hit against an item by 10-15 times.
If a cache.protobuf file is present in the digital resource folder, it is quickly deserialized into the digital resource transfer object, with huge savings in performance.
If the cache.protobuf file is NOT present, the transfer object is built traditionally through the full METS, database data, and transfer object mapping, and the resulting object is then serialized into the cache.protobuf file.
The Builder also maintains this cache.protobuf file for new items and items flagged for additional work.
A system administrator can invalidate these cached metadata files through the system settings screen.
Plugins can include a flag indicating if they invalidate the cached metadata files when enabled or disabled.
A small app is included for developers to run against your instance and pre-cache all of its resources.
New Features
Archival System
The SobekCM application now includes an integrated archival system, including reporting on which files were archived and to which cloud storage systems, as well as checksum data to help ensure the integrity of archived files when they are requested again in the future. The system also stores the MIME type and additional encoding information, so that a migration strategy can be planned for the archived files over their long life.
New archival system database structure and stored procedures
New viewer to allow authenticated users to view the archival information for their resource, and ultimately request files from the archive
Updated the builder to set aside files for archiving, so that any archival tool can retrieve them
Single Sign-On
This new version supports OpenID (which Google uses) as well as SAML for authentication. In addition, the default local authentication system can be turned off completely. Access to the digital library now passes through an extensible, modularized authentication system, so additional authentication systems can also be added via plug-ins.
IIIF Manifest and Search Support
Your digital resources can be exposed with IIIF manifests through new built-in IIIF standard Presentation API and Content Search API implementations. This makes it easier to share your resources with other partner libraries, museums, and archives internationally.
Architectural Flexibility
Two architectural changes allow for more flexibility in how SobekCM is implemented.
SobekCM now supports PostgreSQL and RDS databases, as well as the original MS SQL databases.
Digital resource files can now be placed in GCS cloud buckets, rather than being stored in attached local storage. This should also make implementing more complex access rights easier in the near future, as access to each file is granted temporarily via a signed URL.
Since the core of the web application is now running as a .NET 10 app through Kestrel (and IIS on Windows), this opens up the possibility of running SobekCM on Linux servers as well.
UI Localization
Public UI strings are now driven by localization config files, allowing users to customize strings, even in their own language
Language configuration file determines which languages are supported for collection and web skin resources
Completed the translation of the UI into French and Spanish
Added support and translations for German, Italian, and Dutch. Support for these languages will likely be provided through extensions included in all instances.
Can now easily add new language support as plug-ins for the web application
Restrict Item to User Group
The system now supports restricting access to items by user group.
Removed Restricted_ItemViewer as a distinct item viewer, since the restricted message can now be customized in the system and a separate viewer is no longer necessary
Added a new database and solr field for the Restriction Message
Added a new table for user group access and user access controls on individual items
Added new logic to determine if a user has access to a user group restricted item
If a user has access to an item, a banner is still shown to let the user know the item isn't completely public
Updated results viewers to include code for restricted item messages in search results
Other New Features
Item Management
Can now load a 'mainthm.jpg' file in the page upload screen, which will be used as the main thumbnail
When showing the history of work on an item, if a user ID is present, it now links to the user view
If the system is not using tracking sheets, the option no longer appears in the mySobek home menu
Aggregation Management
Can now set aggregation home pages to show a browse of items below the home page text
Item Count (by Aggregation) now groups items by visibility
Can now use an include directive in home pages to pull in html from other sources without using javascript and multiple HTTP calls
Administrative Features
Added new User Admin role
The permissions brief mapper is now included in the brief item mapping, providing groups and permissions to the UI
General UI
Improved paging on the Private Items list by adding paging buttons to the bottom of the list
Usage stats definitions page now allows the system name to be replaced
Default usage stats definitions source page no longer contains UFDC language
New user settings possible
User settings are now more flexible; keys are defined as strings in the User_Setting_Constants class
Saving a user now also saves their user settings at the same time
Flag indicates whether a user should be shown the QC tool
Flag indicates whether a user should see item behaviors
Can now set users so they cannot change the visibility of their items
New system-wide settings to simplify UI
System-wide settings added to database and Settings.Resources
Flag to suppress the mass behavior from being displayed (defaults to suppressed)
Flag to suppress managing the geo-spatial beta interface (defaults to suppressed)
Flag to suppress all tracking sheet functionality (defaults to suppressed)
OER Support
When a user is set as an instructor in an OER-type collection, QC and behaviors are automatically disabled from being shown
Added new Course Title, Licensing, and Accessibility metadata for digital resources
During registration, if the user's organization matches an institution collection, the org code can be auto-assigned
Added new OpenPublisher ability to create simple OER materials directly in the web interface
Added new OpenTextbook viewer to view the OER
Broke the registration screen away from the preferences viewer, allowing custom registration screens, such as for OER submitters
Can now easily email a submitter when their private items are made public
Metadata Template Changes
The template used is now saved in the history entry for a new item
When setting templates for a user, the name now shows rather than the code
Can now add a prompt to the template XML file, which will appear during editing using that template
Can also add a help URL to the template XML file, which will be used as the link in the text at the top of the edit form
Text area elements now support placeholder text as well
Can now customize the message shown to users after submitting material in the template
Added new controlled target audience template element to allow users to select from a list of target audiences
Publication place element now accepts a default value in the template
UserID is now saved when saving a new item/group from the web
Added support for OpenTelemetry using a local otel collector
Added a health (/health) endpoint to report basic health of the app
Added a new Audio "viewer" for items
Search Changes
Solr Searching
The old Legacy searching system was removed and all searching now goes through Solr
Changed map searching to use Solr indexes as well, which required a Solr schema change.
Solr is used to retrieve the metadata information for database-driven displays, such as users' bookshelves and lists of submitted items. The list of items is pulled from the database, and the metadata is then collected from Solr.
Migrated off the legacy SolrNet library. Now the applications send and parse JSON directly to Solr/Jetty. This allows Solr to be upgraded to the latest version, currently Solr 10
Basic authentication in the solr server is now supported
Updated the solr schema and config to work with the latest Solr version (Solr 10)
Database Metadata System
As such, all the metadata tables have been removed from the database schema, and many stored procedures were either removed or simplified. In addition, full text indexing is no longer needed within the database.
This also made it simpler to add support for RDS and PostgreSQL.
Tech Upgrades & Tech Debt
This version includes the largest set of tech upgrades since SobekCM was originally launched. The highlight is upgrading from .NET Framework 4.7.2 to running on .NET 10, which brought with it a good set of other changes needed.
Hosting model moved from IIS/System.Web model to .NET 10 ASP.NET Core through Kestrel
Elimination of HttpContext.Current required changes throughout the writer/viewer hierarchy.
Caching redesign, from request-specific caching to session caching to application storage
But this version includes more than just the update to the .NET framework; it also included numerous smaller upgrades and general tech debt.
Migrated old DLL libraries to newer NuGet packages and patched any security vulnerabilities
Ran extensive OWASP security scans, mitigating any issues discovered
Vastly simplified how writers and viewers write to the response stream
Dissolved monolithic SobekCM_Page_Globals into QueryInitializer with QueryInitializerHelpers
General dead code/warnings sweep
Removed unused stored procedures from database
Removal of dead, unused calls from the database gateway classes
Refactored the most complex stored procedures into set-based STRING_AGG queries
Resulted in a smaller, more accurate stored-procedure surface
Builder Updates
Builder will now convert JPEG2000s to non-master TIFFs for use in image generation and OCR
New modules added to the builder item processing workflow to remove the non-master TIFFs before archiving
Cleaned up several modules for increased separation of concerns/responsibilities
OCR with Tesseract can now be set to run multi-threaded to speed up OCR processing
Increased logging by the builder of incoming files and processing
Can set a flag when calling the Builder, to have it just run once and stop
Bug Fixes
Single aggregation admin screen should now save correctly and close when done
Print view works correctly again
Share button from individual items also works correctly now
Corrected an issue that prevented Bookshelf/Submitted Items from displaying correctly
Corrected an issue where the left nav bar did not free up space when hidden and unused in the item viewer
The table of contents viewer would throw a fatal exception if the viewer code was not extracted correctly. Corrected.
Thumbnail viewer corrected to handle null PubDate values correctly
Corrected misspelling in the base citation configuration file
If there are no stats for the item, a special message is displayed
Corrected edge case errors for a repository with no collections or items