Skip to content

Confidence interval feature #639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 18, 2024
Merged

Confidence interval feature #639

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
dece867
add confidence intervals
a-kore Jun 13, 2024
e2a5106
fix formatting
a-kore Jun 17, 2024
534f857
Merge branch 'main' into confidence_interval_feature
amrit110 Jun 18, 2024
7991202
add tooltips
a-kore Jun 18, 2024
f44d842
Update requests and imbalanced-learn to fix compatibility issues
amrit110 Jun 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions cyclops/report/model_card/sections.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,16 @@
class Overview(BaseModelCardSection):
"""Overview section with aggregate metrics."""

last_n_evals: Optional[int] = Field(
None,
last_n_evals: int = Field(
0,
description="The number of evaluations to display in the model card.",
)

mean_std_min_evals: int = Field(
3,
description="The minimum number of evaluations required to calculate mean and std in performance over time plot.",
)

metric_cards: Optional[MetricCardCollection] = Field(
None,
description="Comparative metrics between baseline and periodic report.",
Expand Down
6 changes: 6 additions & 0 deletions cyclops/report/report.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1054,6 +1054,7 @@ def export(
interactive: bool = True,
save_json: bool = True,
last_n_evals: Optional[int] = None,
mean_std_min_evals: int = 3,
synthetic_timestamp: Optional[str] = None,
) -> str:
"""Export the model card report to an HTML file.
Expand All @@ -1073,6 +1074,10 @@ def export(
last_n_evals : int, optional
The number of most recent evaluations to include in the report and
calculate trends for. If not provided, all evaluations will be included.
mean_std_min_evals : int
The minimum number of evaluations required to calculate the mean and
standard deviation for the performance over time plot in the overview
section. The default is 3.
synthetic_timestamp : str, optional
A synthetic timestamp to use for the report. This is useful for
generating back-dated reports. The default is None, which uses the
Expand Down Expand Up @@ -1143,6 +1148,7 @@ def export(
if self._model_card.overview is not None:
last_n_evals = 0 if last_n_evals is None else last_n_evals
self._model_card.overview.last_n_evals = last_n_evals
self._model_card.overview.mean_std_min_evals = mean_std_min_evals

self._validate()
template = self._get_jinja_template(template_path=template_path)
Expand Down
22 changes: 21 additions & 1 deletion cyclops/report/templates/model_report/macros.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,27 @@
</div>
<div class="column" style="float: left;">
<div class="subcard_overview" style="padding:10px; margin-bottom:20px; width:15vw;">
<h4>Metrics</h4>
<div style="display:flex; flex-direction:column; gap:10px;">
<h4 style="margin-bottom: 0px;">Metrics</h4>
<div class="radio-buttons" id="mean-std-selection" style="margin-bottom:30px;">
<input type="checkbox" id="mean" value="Mean", name="Mean">
<div class="tooltip">
<label for="mean">Mean</label>
<div class="tooltiptext">
The moving average of all data points.
<div class="arrow-up"></div>
</div>
</div>
<input type="checkbox" id="std" value="Std", name="Std">
<div class="tooltip">
<label for="std">Std</label>
<div class="tooltiptext">
A measure of how dispersed the data points are in relation to the mean.
<div class="arrow-up"></div>
</div>
</div>
</div>
</div>
<div class="radio-buttons" id="slice-selection">
<input type="radio" id="{{comp.metric_cards.metrics[0]}}" name="metric" value="{{comp.metric_cards.metrics[0]}}" checked>
{% if comp.metric_cards.metrics[0]|regex_search("\((.*?)\)")|length != 0 %}
Expand Down
49 changes: 44 additions & 5 deletions cyclops/report/templates/model_report/model_report.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,10 @@

<html lang="en">
<head>
{# <meta id="slices" data-slices="{{ get_slices(model_card)|safe }}" /> #}

<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial_scale=1">
{# <link href="css/bootstrap.min.css" rel="stylesheet"> #}
{# <link href="https://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css" rel="stylesheet"> #}
{# <script src="js/bootstrap.bundle.min.js"></script> #}
{# <script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/js/bootstrap.bundle.min.js"></script> #}

<style>
html {
scroll-behavior: smooth;
Expand Down Expand Up @@ -356,10 +353,23 @@
align-items: center;
}

.radio-buttons {
display: flex;
row-gap: 15px;
flex-direction: row;
flex-wrap: wrap;
justify-content: left;
align-items: center;
}

input[type="radio"] {
display: none;
}

input[type="checkbox"] {
display: none;
}

.radio-buttons label {
font-size:14px;
cursor: pointer;
Expand Down Expand Up @@ -417,6 +427,17 @@
color: #0073e4;
}

.radio-buttons input[type="checkbox"]:checked+label {
background-color: rgba(0, 115, 228, 0.2);
border: 2px solid #0073e4;
color: #0073e4;
}

.radio-buttons input[type="checkbox"]:checked + .tooltip label {
background-color: rgba(0, 115, 228, 0.2);
border: 2px solid #0073e4;
color: #0073e4;
}

.radio-buttons #button {
padding-right: 5px;
Expand Down Expand Up @@ -589,4 +610,22 @@
}
}
document.addEventListener('DOMContentLoaded', updateLastNEvals);

function storeMeanStdMinEvals() {
var mean_radio_button = document.getElementById("mean");
var mean_std_min_evals = {{ model_card.overview.mean_std_min_evals }};
mean_radio_button.value = mean_std_min_evals;
}
document.addEventListener('DOMContentLoaded', storeMeanStdMinEvals);


function changeMeanStd() {
updatePlot();
}

var mean_std_plot_selection = document.querySelectorAll('#mean-std-selection input[type="checkbox"]');
for (let selection of mean_std_plot_selection) {
selection.addEventListener('change', changeMeanStd);
}

</script>
Loading