Update S3 data architecture and integration

[gsvgit]

Some topics for discussion from #63 .

• Can we integrate data description into the file? Does CSV supports comments? Can we use a custom CSV-based format? Something similar to mtx, for example. Or just reuse mtx.
• I'm not sure that the simplicity of CSV is a problem.
• Why we should store plain CSV without compression? I hope, I understand the sentence "It may take more disk space than .tar.gz archives" correctly.

[vadyushkins]

• No. By definition, CSV does not support comments. The only way to add comments is to add the appropriate field to the CSV itself.
• No, the simplicity of the CSV is not such a big problem. But it does not allow expressing complex data associated with nodes and edges.
• Agree. I did not think about the fact that it is possible to store CSV files compressed into separate .tar.gz archives. I think it's worth comparing how quickly the loading of the CSV and the CSV compressed into the .tar.gz archive proceeds. And how much disk space is taken up in each case.
• I think for more complex graphs, we can use one of the formats presented here. I would especially pay attention to this format.

[gsvgit]

Well. So, what about mtx (matrix market) format? This format is used in SuiteSparse matrix collection (https://sparse.tamu.edu/HB/ash331) I think, we should use the similar structure of file header. We should only check that it is possible to use strings as labels in this format.

[vadyushkins]

Unfortunately Matrix Market does not support string entries 😞

I think the best option would be to use a CSV file for storing the graph and a corresponding MarkDown file for storing the graph description.

• I suggest using the following header for the CSV file: [edge_tail, edge_head, edge_label] (like in Wiki)(it's worth discussing)

• I also suggest writing the following information to the corresponding MarkDown file (it's worth discussing):

  • Name -- a name of the graph
  • Version -- a version of the graph
  • Link Download -- a link to download the corresponding CSV graph file
  • Link Origin -- a link to download the original (RDF) graph file
  • Num Nodes -- a number of graph nodes
  • Num Edges -- a number of graph edges
  • Stat Edges -- statistics on graph edges; table [Edge type/ the number of edges of this type]