Update docs for external licenses

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Author: AyanSinhaMahapatra

docs/source/cli-reference/core-options.rst

@@ -69,74 +69,6 @@ Comparing Progress Message Options
 
 ----
 
-``--reindex-licenses`` Option
------------------------------
-
-    ScanCode maintains a license index to search for and detect licenses. When Scancode is
-    configured for the first time, a license index is built and used in every scan thereafter.
-
-    This ``--reindex-licenses`` option rebuilds the license index. Running a scan with this option
-    displays the following message to the terminal in addition to what it normally shows::
-
-        Checking and rebuilding the license index...
-
-    ..
-        [ToDo] Research and Write Better
-
-``--additional-license-directory`` Option
------------------------------------------
-
-    .. admonition:: Dependency
-
-        The option ``--additional-license-directory`` requires the option ``-reindex--licenses``.
-
-    The ``--additional-license-directory`` option allows the user to include additional directories
-    of licenses to use in license detection.
-
-    This command only needs to be run once for each set of additional directories; in all subsequent
-    runs of Scancode with the same directories all the licenses in the directories will be cached.
-
-    The directory structure should look something like this::
-
-        licenses/
-        ├── privateLicense1/
-        │   ├── license/
-        │   │   ├── privateLicense1.LICENSE
-        │   │   └── privateLicense1.yml
-        │   └── rule/
-        │       ├── privateLicense1.RULE
-        │       └── privateLicense1.yml
-        └── privateLicense2/
-            ├── license/
-            │   ├── privateLicense2.LICENSE
-            │   └── privateLicense2.yml
-            └── rule/
-                ├── privateLicense2.RULE
-                └── privateLicense2.yml
-
-    Here is an example of reindexing the license cache using the ``--additional-license-directory PATH`` option with a single directory.
-    Note that ``--reindex-licenses`` **must** come after ``--additional-license-directory``::
-
-        scancode --additional-license-directory /home/user/external_licenses/license1 --reindex-licenses
-
-    You can also include multiple directories like so::
-
-        scancode --additional-license-directory /home/user/external_licenses/external1 --additional-license-directory /home/user/external_licenses/external2 --reindex-licenses
-
-    If you want to continue running scans with ``/home/user/external_licenses/external1`` and ``/home/user/external_licenses/external2``,
-    you can simply run scans after reindexing with those directories and they will be included. ::
-
-        scancode -clpieu --json-pp output.json samples
-
-    However, if you wanted to run a scan with a new set of directories, such as ``home/user/external_licenses/external1``
-    and ``home/user/external_licenses/external3``, you would need to reindex the license index with those directories as parameters. ::
-
-        scancode --additional-license-directory /home/user/external_licenses/external1 --additional-license-directory /home/user/external_licenses/external3 --reindex-licenses
-
-    ..
-
-----
-
 ``--from-json`` Option
 ----------------------
 

docs/source/cli-reference/index.rst

@@ -8,6 +8,7 @@
    help-text-options
    list-options
    simple-examples
+   other-commands
    basic-options
    core-options
    output-format

docs/source/cli-reference/list-options.rst

@@ -30,6 +30,10 @@ available in the command line.
 
 ----
 
+.. include::  /rst_snippets/scancode-reindex-licenses.rst
+
+----
+
 .. include::  /rst_snippets/core_options.rst
 
 ----

docs/source/cli-reference/other-commands.rst

@@ -0,0 +1,102 @@
+Other available CLIs
+====================
+
+.. _other_cli:
+
+----
+
+.. include::  /rst_snippets/scancode-reindex-licenses.rst
+
+----
+
+.. include::  /rst_snippets/extract.rst
+
+----
+
+``scancode-reindex-licenses`` command
+-------------------------------------
+
+ScanCode maintains a license index to search for and detect licenses. When Scancode is
+configured for the first time, a license index is built and used in every scan thereafter.
+
+This ``scancode-reindex-licenses`` command rebuilds the license index. Running this command
+displays the following message to the terminal::
+
+    Checking and rebuilding the license index...
+
+This has several CLI options as follows:
+
+``--additional-directory`` Option:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``--additional-directory`` option allows the user to include additional directories
+of licenses to use in license detection.
+
+This command only needs to be run once for each set of additional directories, in all subsequent
+runs of Scancode with the same directories all the licenses in the directories will be cached
+and used in License detection. But reindexing removes these directories, if they aren't
+reintroduced as additional directories.
+
+The directory structure should look something like this::
+
+    additional_license_directory/
+    ├── licenses/
+    │   ├── example-installed-1.LICENSE
+    │   └── example-installed-1.yaml
+    ├── rules/
+    │   ├── example-installed-1.RULE
+    │   └── example-installed-1.yaml
+
+Here is an example of reindexing the license cache using the ``--additional-directory PATH`` option
+with a single directory::
+
+    scancode-reindex-licenses --additional-directory tests/licensedcode/data/additional_licenses/additional_dir/
+
+You can also include multiple directories like so::
+
+    scancode-reindex-licenses --additional-directory /home/user/external_licenses/external1 --additional-directory /home/user/external_licenses/external2
+
+If you want to continue running scans with ``/home/user/external_licenses/external1`` and
+``/home/user/external_licenses/external2``, you can simply run scans after the command above
+reindexing with those directories and they will be included. ::
+
+    scancode -l --license-text --json-pp output.json samples
+
+However, if you wanted to run a scan with a new set of directories, such as
+``home/user/external_licenses/external1`` and ``home/user/external_licenses/external3``, you would
+need to reindex the license index with those directories as parameters::
+
+    scancode --additional-directory /home/user/external_licenses/external1 --additional-directory /home/user/external_licenses/external3
+
+.. include::  /rst_snippets/note_snippets/additional_directory_is_temp.rst
+
+
+.. note::
+
+    You can also install external licenses through a plugin for
+    better reproducibility and distribution of those license/rules
+    for use in conjunction with scancode-toolkit licenses.
+    See :ref:`install_new_license_plugin`
+
+
+``--only-builtin`` Option:
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Rebuild the license index excluding any additional license directory or additional
+license plugins which were added previously, i.e. with only builtin scancode license and rules.
+
+This is applicable when there are additional license plugins installed already and you want to
+reindex the licenses without these licenses from the additional plugins.
+
+.. note::
+
+    Running the ``--only-builtin`` command won't get rid of the installed license plugins, it
+    would just reindex without the licenses from these plugins for once. Another reindex afterwards
+    without this option would bring back the licenses from the plugins again in the index.
+
+
+``--all-languages`` Option:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Rebuild the license index including texts all languages (and not only
+English) and exit. This is an EXPERIMENTAL option.

docs/source/how-to-guides/install_new_license_plugin.rst

@@ -1,18 +1,25 @@
-.. _install_new_license_plugin:
+.. _install_external_licenses:
+
+How to Install External Licenses to Use in License Detection
+============================================================
 
-How to Install External License Plugins to Use in License Detection
-===================================================================
+Users can install external licenses and rules in the form of:
 
-Users can install external licenses and rules in the form of plugins. These
-licenses and rules are then used in license detection.
+1. reusable plugins
+2. license directories
 
-How to create a plugin containing external licenses and/or rules
-----------------------------------------------------------------
+These licenses and rules are then used in license detection.
+
+.. _install_new_license_plugin:
+
+How to install a plugin containing external licenses and/or rules
+-----------------------------------------------------------------
 
 To create a plugin with external licenses or rules, we must create a Python package
 containing the license and/or rule files. Python packages can have many different
-file structures. You can find an example package in
-``tests/licensedcode/data/example_external_licenses/licenses_to_install1``.
+file structures. You can find an example package in:
+
+``tests/licensedcode/data/additional_licenses/additional_plugin_1``.
 
 This is the basic structure of the example plugin::
 
@@ -26,14 +33,11 @@ This is the basic structure of the example plugin::
     │       │   ├── example-installed-1.RULE
     │       │   └── example-installed-1.yaml
     │       └── __init__.py
-    ├── gpl-1.0.LICENSE
+    ├── apache-2.0.LICENSE
     ├── MANIFEST.in
     ├── setup.cfg
     └── setup.py
 
-Key points to note
-------------------
-
 Entry points definition in ``setup.py``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -75,18 +79,27 @@ After creating this plugin, you can upload it to PyPI so that others can use it,
 leave it as a local directory.
 
 Installing and using the plugin
--------------------------------
-To use the plugin in license detection, all you need to do is install it using ``pip``.
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To use the plugin in license detection, all you need to do is:
+
+1. Configure the scancode-toolkit virtualenv and activate.
+2. Install the package with `pip` like the following:
+   ``pip install tests/licensedcode/data/additional_licenses/additional_plugin_2/``
+3. Reindex licenses using `scancode-reindex-licenses`.
+
+.. include::  /rst_snippets/note_snippets/license_plugin_needs_reindex.rst
+
 Once it is installed, the contained licenses and rules will automatically be used in
 license detection assuming the plugin follows the correct directory structure conventions.
 
 Writing tests for new installed licenses
-----------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Look at ``tests/licensedcode/data/example_external_licenses/licenses_to_install1`` to see
 an example of a plugin with tests. The tests are contained in the ``tests`` directory::
 
-   licenses_to_install1/
+    licenses_to_install1/
     ├── src/
     │   └── licenses_to_install1/
     │       ├── licenses/
@@ -132,8 +145,44 @@ Then you can define a test class and call the ``build_tests`` method defined in
 The ``tests/data`` directory contains a pair of files for each license:
 a license text file and a YAML file specifying the expected license expressions from the test.
 
-Finally, to run the test, do the following:
+Finally, install the plugin and run the test:
+
+``pytest -vvs tests/test_detection_datadriven.py``.
+
+.. include::  /rst_snippets/note_snippets/license_plugin_delete.rst
+
+----
+
+.. _add_new_license_directory:
+
+How to add external licenses and/or rules from a directory
+----------------------------------------------------------
+
+This is the basic structure of the example license directory::
+
+    additional_license_directory/
+    ├── licenses/
+    │   ├── example-installed-1.LICENSE
+    │   └── example-installed-1.yaml
+    ├── rules/
+    │   ├── example-installed-1.RULE
+    │   └── example-installed-1.yaml
+
+Adding the licenses to the index
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To add the licenses in the directory to the index, all you need to do is:
+
+1. Configure the scancode-toolkit virtualenv and activate.
+2. Run ``scancode-reindex-licenses`` with:
+
+   ``--additional-directory tests/licensedcode/data/additional_licenses/additional_dir/``
+
+.. include::  /rst_snippets/note_snippets/additional_directory_is_temp.rst
+
+
+Once the licenses/rules are in the index, they will automatically be used in license detection.
+
+----
 
-1. Create a virtual environment to install the package into.
-2. Install the package using ``pip``, e.g. ``pip install ./licenses_to_install1``.
-3. Run the tests, e.g. ``py.test tests/test_detection_datadriven.py``.
+.. include::  /rst_snippets/scancode-reindex-licenses.rst

docs/source/rst_snippets/core_options.rst

@@ -11,14 +11,6 @@ All "Core" Scan Options
 --timeout FLOAT          Stop scanning a file if scanning takes longer
                          than a timeout in seconds.  [Default: 120]
 
---reindex-licenses       Force a check and possible reindexing of the
-                         cached license index.
-
---additional-license-directory PATH
-
-          Include paths to directories containing additional licenses and rules to use
-          in license detection. This can be used multiple times for multiple directories.
-
 --from-json              Load codebase from an existing JSON scan
 
 --max-in-memory INTEGER  Maximum number of files and directories scan
@@ -33,5 +25,3 @@ All "Core" Scan Options
                          including and below the starting point. INTEGER
                          must be positive or zero for no limit.
                          [Default: 0]
-
-.. include::  /rst_snippets/note_snippets/core_indep.rst

docs/source/rst_snippets/note_snippets/additional_directory_is_temp.rst

@@ -0,0 +1,6 @@
+.. note::
+
+    Adding licenses/rules from an additional directory is not permanent.
+    Another reindexing without the additional directory option would
+    just use the builtin scancode licenses and rules, and will not have
+    these additonal licenses/rules anymore.

docs/source/rst_snippets/note_snippets/core_indep.rst

@@ -0,0 +1,7 @@
+.. note::
+
+    Once you install a external license plugin, you have to reconfigure
+    scancode-toolkit (or use pip uninstall) to uninstall the plugin to
+    completely remove it. Otherwise using the --only-builtin option only
+    regenerates the index without the installed plugins, but another Reindex
+    would have the licenses/rules from the installed plugins.

docs/source/rst_snippets/note_snippets/license_plugin_delete.rst

@@ -0,0 +1,5 @@
+.. note::
+
+    Installing the plugin will not add the licenses/rules to the
+    index automatically, they will be indexed only after running
+    `scancode-reindex-licenses`.