GNIP 97: New metadata editor

[etj]

GNIP 97 - New metadata editor

Overview

The metadata editor still relies on a legacy template engine which is quite hard to customize.
Also it uses unmaintained vulnerable js libraries that should be removed.

Proposed By

• @etj Emanuele Tajariol

Assigned to Release

This proposal is for GeoNode 5

State

• Under Discussion
• In Progress
• Completed
• Rejected
• Deferred

Motivation

• Allow easier customization
• Allow different editing frontends
• Make a clear distincton between descriptive metadata and operational metadata (owner, group)
• Remove old unmaintained vulnerable js libraries

Proposal

The idea is to switch to standard procedures to document the metadata schema and to return the metadata info.

The metadata schema is presented using the JSON schema conventions.

The GeoNode API has been extended to provide

• /api/v2/metadata/schema metadata schema
• /api/v2/metadata/instance/<PK> resource metadata instances handling (GET/PUT)
• /api/v2/metadata/xxx other utility calls for the metadata handling (autocomplete endpoints referred into the schema, etc)

One of the basic driving features has been the customization of the schema. The implementation relies on a set of metadata handlers that can be easily extended. We embedded into GeoNode the optional inspire module, mostly a (working) showcase that documents how the GeoNode base model can be customized.

Implementation

Intro

The package geonode.metadata contains all the metadata code.

The manager module contains the base engine that calls methods on the registered field handlers.
Handlers are the objects that manage the field definition (the json schema), and that take care of converting database objects into json schema instances and viceversa.
A single handler may handle one or more fields.

MetadataManager

The geonode.metadata.manager.MetadataManager class provides these main methods:

  def get_schema(self, lang=None)`

Builds and returns the metadata json schema.
The schema is created by looping on the sorted set of metadata handlers, asking them to update the schema incrementally via def update_schema(self, jsonschema: dict, context, lang=None).
Handlers may add their own field, remove fields added by previously called
The schema localization is only related to the title and description annotations

This method is used by the /api/v2/metadata/schema endpoint.
The default language used is the one set in the cookies, or it can be overriden by adding the query param lang.

   def build_schema_instance(self, resource: ResourceBase, lang=None)

Builds and returns the json schema instance related to the resource.

The instance is created by examining the json schema and looping on each field.
Then, the related handler is called to fill in the serialized value for such field.

Called handler methods:

• def load_serialization_context(resource: ResourceBase, jsonschema: dict, context: dict)
  Allow handlers to pre-load some common info for all the handled fields
• def get_jsonschema_instance(resource: ResourceBase, field_name, context, errors, lang=None)
  Returns the instance of the sub-schema associated with the field field_name.

   def update_schema_instance(self, resource, json_instance)

Parses the incoming json schema instance and stores the value into the DB.

Persistence is done by examining the json schema and looping on each field.
Then, the related handler is called to parse and store the deserialized value(s) for such field.
If the field is a complex object, it's up to the handler parse and handle it.

Called handler methods:

• def load_deserialization_context(self, resource: ResourceBase, jsonschema: dict, context: dict)
  Allow handlers to pre-load some common info for all the handled fields
• def update_resource(resource: ResourceBase, field_name: str, json_instance: dict, context: dict, errors: dict, **kwargs)
  Logic to parse and persist the field

Handlers

The list of handlers is declared in geonode.metadata.settings.METADATA_HANDLERS.
Order is important since later declared handlers may want to customize previously defined fields.
Within each field subschema there are some geonode-specific annotations (custom annotation are allowed by the json schema standard), some are needed for backend logic (for instance geonode:handler'), other for the client UI (e.g. ui:options`)

BaseHandler

BaseHandler handles most of the fields declared inside the ResourceBase class.
The BaseHandler will read the main schema from a file (metadata/schemas/base.json ) , augmenting its info programmatically (such as autocomplete URLs).

SparseHandler

This handler is not actively used by the default GeoNode schema, since it's more aimed at customizations.
You may see an example of its use in the inspire package.

The idea for the SparseField (the single model change in metadata) is to have a "vertical" table with free form fields:

In the default case, its use is as simple as declaring the field:

  "instrumentations": {
    "type": "array",
    "title": "instrumentations",
    "description": "Provides information about any instruments used in the data collection. The description should include vendor, model number, optional equipment, etc.",
    "items": {
      "type": "string",
      "minLength": 1,
      "default": ""
    },
    "minItems": 1,
    "geonode:handler": "sparse"
  }

Some custom annotation

• geonode:handler: key to the handlers defined in METADATA_HANDLERS; tells which is the handler to be called to handle the field's content. Pls note that this annotation may be replaced by subsequent handler if they want to handle the field differently.
• geonode:required: the jsonschema specs wants the required array outside the subschema definition. By defining this annotation as true, the metadata manager will add the current field in the required list.

Localization

So far the static labels for metadata editor have been localized using gettext() and i18n .po files.

Since the jsonschema is much more dynamic in its behaviour, allowing very custom schema, we need other ways for localizing the schema fields labels.

The proposed/implemented idea is to use a thesaurus with a well known identifier labels_i18n where

• Thesaurus:
  • identifier = "labels_i18n"
• ThesaurusKeyword:
  • about = field name
  • alt_label= field name
• ThesaurusKeywordLabel:
  • localized stuff

In order to avoid clashes, the schema fields titles should be prefixed with the metadata handler id ("inspire", "cnr", ...)

The implementation will look for translations in the thesaurus, and then will the gettext as fallback.

Some improvements to the thesauri manament commands will be done in order to better manage updates.
For instance, we may want to

• merge a thesaurus file into an existing thesaurus
• import from a thesaurus file only a subset of the keywords
• ...

Backwards Compatibility

TODO

Future evolution

This refactoring allows for other major improvemetns in GeoNode.
Since this is quite a big improvement in GeoNode that introduces breaking changes, the plans before releasing a stable 5.0 include some other improvements/refactoring, that will be discussed each in its own issue:

• contact roles refactoring
  • improve the model allowing non-system users to be contact points
  • easier customization
• codelists into thesauri
  • remove many custom codelist tables
  • improve localization
• move owner and group outside the metadata editor, since they are not informative metadata, but operative info, and should be handler in a differet localtion and with different privileges. At the moment we don't want the metadata schema to chhange according to user privileges.

Most of the pure metadata fields in ResourceBase can be moved as SparseFields, improving the overall performance of the ORM procedures.

Feedback

TODO

Voting

Project Steering Committee:

• Alessio Fabiani:
• Francesco Bartoli:
• Giovanni Allegri:
• Simone Dalmasso:
• Toni Schoenbuchner:
• Florian Hoedt:

Links

Remove unused links below.

• Pull Request: GNIP 97: New metadata editor #12794
• Linked Issue

[etj]

Some comments from another issue referencing this topic:

• GNIP 96: Implement relations between resources #11494 (comment) by @giohappy

  @gannebamm at the moment the idea is just to implement the simple relationship model. We can extend it to include additional properties in the future, but this will require also extending the editor. This is something I wouldn't do until the metadata editor has not been migrated to the new client. I have almost completed the design of a proposal for the new metadata editor, but no timeline at the moment since.

• GNIP 96: Implement relations between resources #11494 (comment) by @gannebamm

  Thünen is currently looking at ISO-19115 metadata fields not yet covered in the GeoNode metadata schema. We will need to extend the wizard. Also, complete ISO contact roles per ressource base with multiplicity #10290 // [Fixes #10290] complete ISO contact roles per ressource base with multiplicity #10367 will require changes in the wizard. If possible, we could try to work together on the new metadata editor, and you could get some feedback on its developer experience. From our side, for example, @ahmdthr and @ridoo are part of the development team. What do you think?

[mwallschlaeger]

Thanks for the proposal @gannebamm . I just wanted to add some of my thoughts on this topic to the discussion. First of all I want to give some impression on the current interaction with metadata inside of geonode. As geonode ist a project that is growing for long time it has a long history of internal APIs. In the current implementation the metadata values are accessed through Django methods like Querrysets, and Forms and since some time also by the django-rest-api including their serializers and Viewsets. Therefore when adding new fields to the metadata model the following parts need editing:

• rest v2 API (serializer, Viewset)
• Django Forms need to be created in order to edit the metadata in the metadata editor
• and if somewhere else metadata values got accessed they can use querrysets
• django templates

This different methods also require different validation strategies which require some maintenance when editing the metadata model.

Therefore, as we now have a more or less robust rest API, I would suggest to only use the API for interaction with the metadata model. Additionally the base.forms and templates could be removed, and a lot of code needs to be edited to make metadata interaction generally API based. But afterwards, there would be place for a new metadata editor only using rest v2 to interact with geonode django-rest-api backend. This would also allow wider possibilities for geonode metadata editors and would make space for geonode project allow different templates for the metadata editor like (slim, medium, ISO19115:2003).

Still if somebody requires more or other fields for metadata, she need to change the database model, rest API and UI and something not mentioned yet, the pyCSW interaction. For the pyCSW interaction I go definitely with the issue #9147 which would make it easier to bring changes to metadata model to the CSW API.

[giohappy]

@gannebamm @mwallschlaeger The high level design I'm planning for the rewrite of the metadata editor is the following:

• implement a model for the configuration of the metadata editor panels that the client will use to build the new editor. This model will include at least the following:
  • fieldsets (that can be rendered as panels, blocks of widgets, tabs, etc.)
  • fieldsets:
    • fields: list of field configurations
    • name: name of the fieldset
  • field configuration:
    • label
    • name
    • widget type (select, textarea, text input, date, etc.)
    • enums: either a fixed list of values or the API endpoint from where the list can be retrieved dynamically (e.g. categories, regions, etc.)
    • cardinality: 0..n, 1, 1..n, etc.
    • advanced: true/false

As you will notice, this configuration is very similar to what we have for the new filters and facets API.

This configuration could be initially serialized (JSON) and served to the client statically. It could be overridden inside a project, similar to the geocode config overrides.
A second step could be to make the configuration build dynamically and define an interface for modules to register additional fieldsets, fields, and field configurations dynamically.

The goal is to:

• make configurations, APIs, and the client for the metadata editor converge on what we already do for filters, avoiding duplicated models and configuration schemas where possible
• make it generic enough to be able to implement other custom forms, beyond the metadata editor
• merge the Wizard / Advanced visualizations into a single editor, with fieldsets organized in collapsible panels (for example) and with togglable advanced fields

Ok, this is a very rough description. I wanted to take more time to write it down, but given the WIP GNIP has been created already I think it was useful to share it right away.

[mwallschlaeger]

@giohappy thanks for sharing this idea with us. I was spinning my head around this for some time the recent days. If i understand you correct this is only related to the UI part, so the JSON would not configure any API endpoints or database columns behind the API right?

I think generally your idea could be a good approach to make it easier to edit metadata editor in geonode. But for some projects i was working on in the past on which I applied the same approach, I ended up with blowing and extending the JSON definition more and more because the definitions are not fine grained enough. And when I think about my latest experiences with the widgets and forms in geonode I guess it's going to be very hard to implement all the aspects of this into a simple JSON model. I still like the idea and approach to make it more easy to edit the metadata editor view.

[gannebamm]

Just a quick mention of some work done by @t-book in that regard: https://github.com/csgis/custom_metadata

[t-book]

This would also mean implementing the metadata editor with React as well, right? If so, I am rather skeptical in terms of complexity. ( Based on experiences with the development with mapstore2 ). But I would wait for the final GNIP.

[gannebamm]

@t-book What other alternatives would be maintainable? One of the goals of @giohappy is

make configurations, APIs, and the client for the metadata editor converge on what we already do for filters, avoiding duplicated models and configuration schemas where possible

Which implies using mapstore2 as the framework.

Even though I also see mapstore2 developing as a daunting and complex task. However, I am no front-end developer; therefore, this is heavily biased and likely due to inexperience.

[ridoo]

@giohappy as far I can see, the API you have in mind would provide client instructions how to render widgets based on metadata which is might be configurable on the backend side.

Do you have any reasons to include such a coupling between REST API and client? When it comes to metadata, I would have expected the client to make sense of the metadata for its own scope (a client does not always render things).

You already have an OpenAPI description available. Did you accounted this already, and why not to extend metadata this way? Even JSON schema might be an option (but I have to admit not to thought about this option deeper, yet).

If "coupling" gets necessary, maybe configurable layouting on the client side can be an option.

[giohappy]

Do you have any reasons to include such a coupling between REST API and client? When it comes to metadata, I would have expected the client to make sense of the metadata for its own scope (a client does not always render things).

It depends on what you mean by coupling. My proposal is to have a way to tell the client what we want to edit, and how to lay it out (given a list of available widgets/components). This configuration must be dynamic because we want projects to extend and/or modify it. Or even, in the future, create custom forms beyond the metadata editor.

Where should these configurations stay? How do you give it to the client? An API is what I would use. I'm not expecting to implement such an API in GeoNode itself but in the geonode-mapstore-client app. This app has already a lot of "coupling" with the client, being the app that renders the client. At the same time, it decouples the client from GeoNode's core.

You already have an OpenAPI description available. Did you accounted this already, and why not to extend metadata this way? Even JSON schema might be an option (but I have to admit not to thought about this option deeper, yet).

Here we're talking of something different. We want a model to describe forms, I don't see how this can be covered by the current API.

This would also mean implementing the metadata editor with React as well, right? If so, I am rather skeptical in terms of complexity. ( Based on experiences with the development with mapstore2 ). But I would wait for the final GNIP.

React doesn't imply MapStore2. We can see if MapStore2's routing and state management can help or not here, but we're not assuming that we MUST use MapStore2.
On the other hand, we promote React2 to avoid using vanilla Javascript for such an important and complex component and to harmonize the client components.
Of course, this can be discussed with the people that will work on this. What we really care is avoid Frankesteins :)

We're still fighting with the plethora of approaches in the legacy templates. Parts written with Angular, others with jQuery + dozens of unmaintained libraries.

[ridoo]

Where should these configurations stay? How do you give it to the client? An API is what I would use. I'm not expecting to implement such an API in GeoNode itself but in the geonode-mapstore-client app. This app has already a lot of "coupling" with the client, being the app that renders the client. At the same time, it decouples the client from GeoNode's core.

Seems, that I thought you wanted to add this to the GeoNode API (/api/v2). In general, I agree, that geonode-mapstore-client has lot of coupling with the UI .. it actually provides the UI (with some GeoNode integration). Here, such API would be better located than in the GeoNode API of course.

However, until now I have thought of the backend implementation of geonode-mapstore-client is a temporary approach on the path to completely separate frontend from backend. Adding REST API functionality in the contrib module here, feels a bit like cheating 😄, but I may not see all the parts playing a role here. Do you have more strategic insights how geonode-mapstore-client will evolve (a link would also be fine)?