[integration] Add support of Graal to ELK & Mordred

[inishchith]

The task involves an initial exploration of ELK & Mordred to look/discuss an approach to integrate Graal with the toolchain.

• ELK chaoss/grimoirelab-elk
  • Initial CoCom Backend integration
• Mordred chaoss/grimoirelab-sirmordred
  • Requires documentation changes in future

[inishchith]

@valeriocos @jgbarah @aswanipranjal

I have had a look at the codebase of ELK and Mordred. And as we had discussed earlier (in the proposal).
I was thinking of adding connectors to ELK for CoCom backend first (which would require a deeper understanding for me and add more backends in the process).

• We can have sections in Mordred as below: (for CoCom)

[cocom]
raw_index = git_cocom
enriched_index = git_cocom_enriched
latest-items = true
category = ...

• The above would require adding more connectors to ELK with a dependency to Graal. Each connector would be composed [a Graal backend, a raw, enrich connector, a Graal command] as shown below:

"cocom": [CoCom, CoComOcean, CoComEnrich, CoComCommand],

Would love to have suggestions on the approach :)

[valeriocos]

@inishchith yes, this is exactly the idea. If you think it can be useful, I can push an example to a branch in my forks of ELK and mordred, what do you think ?

[valeriocos]

@inishchith I have just pushed some code :)

Graal should be enhanced to mimic the way the Git Perceval backend is called from Mordred. In a nutshell, in the projects.json we declare just the url of the Git repository, the local path where the repo will be cloned (git_path) is automatically derived. A PR is available on Graal: chaoss/grimoirelab-graal#31 to tackle this enhancement (however I need to provide tests)

Here you will find some code that kicstarts the integration between graal and ELK. Feel free to reuse this code and improve it.

In order to use that code, you just need to include in the projects.json and setup.cfg the settings below. Then you can launch micro-mordred with the params --raw --enrich --cfg ./setup.cfg --backends cocom

setup.cfg

[cocom]
raw_index = test_cocom_raw
enriched_index = test_cocom

projects.json

"cocom": [
            "https://github.com/chaoss/grimoirelab-perceval"
        ],

[inishchith]

@valeriocos Thanks for sharing this. I've started some work and will update in due time.

Thanks a lot :)

[valeriocos]

You're welcome @inishchith ! :)

[inishchith]

@valeriocos After getting the initial work executed and tested locally, I had some thoughts over the improvements that could be worked on.

• As we can see (above) that the analysis attribute in case of CoCom enrich index ( being an array of objects [which may contain analysis for multiple files] ) is not well supported for visualization. I was thinking if we can have a discussion over how we can handle such data appropriately and if you can share an idea.

Thanks :)

[valeriocos]

Sure @inishchith ! Note that generally an enriched index contains just flat JSON docs (without nested attributes). The enricher itself takes care of flattening the raw data (or just a part of it). However, changes can be made also in Graal, if needed.

[inishchith]

@valeriocos Thanks for the quick response.

However, changes can be made also in Graal, if needed.

Yes. I'm working on a new structure and evaluating them (in the context of both Graal and ELK). I'll share them once I'm done to get some clarity and suggestions :)

[inishchith]

@valeriocos As mentioned earlier,

The current structure of analysis attribute -

analysis =[ {
  "ccn": 19,
  "avg_ccn": 2.111111111111111,
  "avg_loc": 9.11111111111111,
  "avg_tokens": 64,
  "num_funs": 9,
  "loc": 129,
  "tokens": 786,
  "ext": "py",
  "blanks": 48,
  "comments": 63,
  "file_path": "graal/codecomplexity.py"
}]

As we can see the above data is produced per commit ( and a commit can contains changes made to multiple files )

• As analysis data produced by graal is for every commit. I've made some improvements by pulling out the fields such as ccn, loc,tokens,blanks, comments & file_path to infer the following.

files: total number of files changed in a commit.
total_code_complexity: total code complexity of the files
total_lines_of_code: total lines of code 
total_blank_lines: total blank lines in the files
total_number_of_functions: total number of functions in the files
total_number_of_tokens:  total number of tokens in the files

• There's some work to be done in order to analyze the evolution of each field.

Work Branch - inishchith/grimoirelab-elk#gsoc-graal-2019

Reference Issue: chaoss/metrics#139