Scanner: Storing the scan result via HTTP storage backend fails with WebDAV on a Caddy-based HTTP file server

[AndreasHennig]

The documentation states that storing a scan result using an HTTP storage backend works with "any HTTP file server"
(https://github.com/oss-review-toolkit/ort#http-storage)

I set up a web server using Caddy (https://caddyserver.com/) with the WebDAV module using the following Caddyfile:

:8080 {
  root /var/www/ort-cache
  webdav / {
    scope /var/www/ort-cache/
    modify true
  }
  log stdout
}

The scanner section in my config.yml contains the following definition:

scanner:
    # (...)
    storages:
      http:
        backend:
          httpFileStorage:
            url: 'https://url-to-my-webserver.de'
            headers:
              X-JFrog-Art-Api: "dummy" # See https://github.com/oss-review-toolkit/ort/issues/6758#issuecomment-1486477756

    storageReaders: ["http"]
    storageWriters: ["http"]
    # (...)

I am doing the scan with the following command:

./cli/build/install/ort/bin/ort --info scan --package-types PACKAGE -f JSON -i path/to/analyzer-result.json -o path/to/output/folder

However, storing the scan result fails with the following output:

[main] WARN  org.ossreviewtoolkit.scanner.storages.ProvenanceBasedFileStorage - Could not store scan result for 'RepositoryProvenance(vcsInfo=VcsInfo(type=Git, url=https://github.com/PrismLibrary/Prism.git, revision=15140a61976d0a224cd6ebb9ee1f7ca63db02b47, path=), resolvedRevision=15140a61976d0a224cd6ebb9ee1f7ca63db02b47)' at path 'repository/Git/https%3A%2F%2Fgithub.com%2FPrismLibrary%2FPrism.git/15140a61976d0a224cd6ebb9ee1f7ca63db02b47/scan-results.yml': IOException: Could not store file at 'https://url-to-my-webserver.de/repository/Git/https%3A%2F%2Fgithub.com%2FPrismLibrary%2FPrism.git/15140a61976d0a224cd6ebb9ee1f7ca63db02b47/scan-results.yml': 404 -

So the web server responses with a 404 Not Found error, stating that the looked up resource cannot be found.

This does not seem to be that much of a surprise, since the Scanner tries to store the scan result on the HTTP file server using the PUT method. However, the scan-result.yml apparently gets placed in a folder structure based on the name of the scanned library, in this case /repository/Git/https%3A%2F%2Fgithub.com%2FPrismLibrary%2FPrism.git/15140a61976d0a224cd6ebb9ee1f7ca63db02b47/.

This folder structure does not (yet) exist on the web server. As far as I know, WebDAV does not support "recursive uploading" via PUT. So this folder structure is not created automatically. With WebDAV, a resource first has to be created using the MKCOL method (https://http.dev/webdav).

Do you have a solution or a workaround for this? Is WebDAV supported in the first place?

[sschuberth]

Is WebDAV supported in the first place?

We've never tried 😀 So far the only thing I'm aware of we tried was Artifactory.

So, if you could help here via a code contribution, that'd be appreciated 😉

[sschuberth]

Would probably be nice if #6603 was implemented first.

[sschuberth]

Looks like there's some code in main...brueggenthies-ams:ort:webdav that we could ask @brueggenthies-ams to contribute to fix this 😉

[brueggenthies-adesso]

@sschuberth you are right, that was a just a small experiment on my side, and I am definitely planning on contributing it. I just need to check the contribution guidelines first and probably clean it up a bit.
Will try to do this during this week :)

[sschuberth]

Will try to do this during this week :)

Thanks, but no rush; we're on summer break until mid-August anyway.

[sschuberth]

Closed as part of backlog grooming. Feel free to comment if you would like to contribute to this.