Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/current/v23.2/cluster-setup-troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,7 @@ Like any database system, if you run out of disk space the system will no longer
- [What happens when a node runs out of disk space?]({% link {{ page.version.version }}/operational-faqs.md %}#what-happens-when-a-node-runs-out-of-disk-space)
- [Why is memory usage increasing despite lack of traffic?]({% link {{ page.version.version }}/operational-faqs.md %}#why-is-memory-usage-increasing-despite-lack-of-traffic)
- [Why is disk usage increasing despite lack of writes?]({% link {{ page.version.version }}/operational-faqs.md %}#why-is-disk-usage-increasing-despite-lack-of-writes)
- [Can I reduce or disable the storage of timeseries data?]({% link {{ page.version.version }}/operational-faqs.md %}#can-i-reduce-or-disable-the-storage-of-time-series-data)
- [Can I reduce the storage of timeseries data?]({% link {{ page.version.version }}/operational-faqs.md %}#can-i-reduce-the-storage-of-time-series-data)

###### Automatic ballast files

Expand Down
6 changes: 0 additions & 6 deletions src/current/v23.2/kibana.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,12 +111,6 @@ Click **Refresh**. The query metrics will appear on the dashboard:

<img src="{{ 'images/v23.2/kibana-crdb-dashboard-sql.png' | relative_url }}" alt="CockroachDB Overview dashboard for Metricbeat with SQL metrics" style="border:1px solid #eee;max-width:100%" />

## Step 5. Disable DB Console's local storage of metrics (optional)

If you rely on external tools such as Kibana for storing and visualizing your cluster's time-series metrics, Cockroach Labs recommends that you [disable the DB Console's storage of time-series metrics]({% link {{ page.version.version }}/operational-faqs.md %}#disable-time-series-storage).

When storage of time-series metrics is disabled, the cluster continues to expose its metrics via the [Prometheus endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#prometheus-endpoint). The DB Console stops storing new time-series cluster metrics and eventually deletes historical data. The Metrics dashboards in the DB Console are still available, but their visualizations are blank. This is because the dashboards rely on data that is no longer available. You can create queries, visualizations, and alerts in Kibana based on the data it is collecting from your cluster's Prometheus endpoint.

## See also

- [Monitoring and Alerting]({% link {{ page.version.version }}/monitoring-and-alerting.md %})
Expand Down
6 changes: 0 additions & 6 deletions src/current/v23.2/monitor-cockroachdb-with-prometheus.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,12 +201,6 @@ Although Prometheus lets you graph metrics, [Grafana](https://grafana.com/) is a

1. [Add the dashboards to Grafana](http://docs.grafana.org/reference/export_import/#importing-a-dashboard).

## Step 6. Disable DB Console's local storage of metrics (optional)

If you rely on external tools such as Prometheus for storing and visualizing your cluster's time-series metrics, Cockroach Labs recommends that you [disable the DB Console's storage of time-series metrics]({% link {{ page.version.version }}/operational-faqs.md %}#disable-time-series-storage).

When storage of time-series metrics is disabled, the cluster continues to expose its metrics via the [Prometheus endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#prometheus-endpoint). The DB Console stops storing new time-series cluster metrics and eventually deletes historical data. The Metrics dashboards in the DB Console are still available, but their visualizations are blank. This is because the dashboards rely on data that is no longer available. You can create queries, visualizations, and alerts in Prometheus and AlertManager based on the data Prometheus is collecting from your cluster's Prometheus endpoint.

## See also

- [Monitoring and Alerting]({% link {{ page.version.version }}/monitoring-and-alerting.md %})
Expand Down
8 changes: 1 addition & 7 deletions src/current/v23.2/monitoring-and-alerting.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,7 @@ Each cluster automatically exposes its metrics at an [endpoint in Prometheus for
- Allow you to create and share dashboards, reports, and alerts based on metrics.
- Do not run within the cluster, and can help you to investigate a situation that led up to cluster outage even if the cluster is unavailable.

Metrics collected by the DB Console are stored within the cluster, and the SQL queries that create the reports on the Metrics dashboards also impose load on the cluster. To avoid this additional load, or if you rely on external tools for storing and visualizing your cluster's time-series metrics, Cockroach Labs recommends that you [disable the DB Console's storage of time-series metrics]({% link {{ page.version.version }}/operational-faqs.md %}#disable-time-series-storage).

When storage of time-series metrics is disabled, the cluster continues to expose its metrics via the [Prometheus endpoint](#prometheus-endpoint). The DB Console stops storing new time-series cluster metrics and eventually deletes historical data. The Metrics dashboards in the DB Console are still available, but their visualizations are blank. This is because the dashboards rely on data that is no longer available.
Metrics collected by the DB Console are stored within the cluster, and the SQL queries that create the reports on the Metrics dashboards also impose load on the cluster.

#### SQL Activity pages

Expand Down Expand Up @@ -1001,10 +999,6 @@ Many of the [third-party monitoring integrations]({% link {{ page.version.versio

If you have configured [Prometheus]({% link {{ page.version.version }}/monitor-cockroachdb-with-prometheus.md %}) to monitor your CockroachDB instance, you can also configure alerting rule definitions to have Alertmanager detect [important events](#events-to-alert-on) and alert you when they occur.

If you rely on external tools for storing and visualizing your cluster's time-series metrics, Cockroach Labs recommends that you [disable the DB Console's storage of time-series metrics]({% link {{ page.version.version }}/operational-faqs.md %}#disable-time-series-storage).

When storage of time-series metrics is disabled, the DB Console Metrics dashboards in the DB Console are still available, but their visualizations are blank. This is because the dashboards rely on data that is no longer available.

#### Prometheus alerting rules endpoint

Every CockroachDB node exports an alerting rules template at `http://<host>:<http-port>/api/v2/rules/`. These rule definitions are formatted for easy integration with Alertmanager.
Expand Down
10 changes: 5 additions & 5 deletions src/current/v23.2/operational-faqs.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ For the first 10 days of your cluster's life, you can expect storage per node to

or about 6 GiB. With on-disk compression, the actual disk usage is likely to be about 4 GiB.

However, depending on your usage of time-series charts in the [DB Console]({% link {{ page.version.version }}/ui-overview-dashboard.md %}), you may prefer to reduce the amount of disk used by time-series data. To reduce the amount of time-series data stored, or to disable it altogether, refer to [Can I reduce or disable the storage of time-series data?](#can-i-reduce-or-disable-the-storage-of-time-series-data)
However, depending on your usage of time-series charts in the [DB Console]({% link {{ page.version.version }}/ui-overview-dashboard.md %}), you may prefer to reduce the amount of disk used by time-series data. To reduce the amount of time-series data stored, refer to [Can I reduce the storage of time-series data?](#can-i-reduce-the-storage-of-time-series-data)

## Why is my disk usage not decreasing after deleting data?

Expand Down Expand Up @@ -136,11 +136,11 @@ The CockroachDB `internal-delete-old-sql-stats` process cleans up query executio

In general, the `internal-delete-old-sql-stats` process is not expected to impact cluster performance. There are a few cases where there has been a spike in CPU due to an incredibly large amount of data being processed; however, those cases were resolved through [workload optimizations]({% link {{ page.version.version }}/make-queries-fast.md %}) and general improvements over time.

## Can I reduce or disable the storage of time-series data?
## Can I reduce the storage of time-series data?

Yes, you can either [reduce the interval for time-series storage](#reduce-the-interval-for-time-series-storage) or [disable time-series storage entirely](#disable-time-series-storage).
Yes, you can [reduce the interval for time-series storage](#reduce-the-interval-for-time-series-storage).

After reducing or disabling time-series storage, it can take up to 24 hours for time-series data to be deleted and for the change to be reflected in DB Console metrics.
After reducing time-series storage, it can take up to 24 hours for time-series data to be deleted and for the change to be reflected in DB Console metrics.

### Reduce the interval for time-series storage

Expand Down Expand Up @@ -171,7 +171,7 @@ To reduce the interval for storage of time-series data:

- For data stored at 30-minute resolution, reduce the `timeseries.storage.resolution_30m.ttl` cluster setting to an [`INTERVAL`]({% link {{ page.version.version }}/interval.md %}) value less than `2160h0m0s` (90 days).

Cockroach Labs recommends that you avoid _increasing_ the period of time that DB Console retains time-series metrics. If you need to retain this data for a longer period, consider using a third-party tool such as Prometheus to collect the cluster's metrics and disabling the DB Console's collection of time-series metrics. Refer to [Monitoring and Alerting]({% link {{ page.version.version }}/monitoring-and-alerting.md %}).
Cockroach Labs recommends that you avoid _increasing_ the period of time that DB Console retains time-series metrics. If you need to retain this data for a longer period, consider using a third-party tool such as Prometheus to collect and store metrics. Refer to [Monitoring and Alerting]({% link {{ page.version.version }}/monitoring-and-alerting.md %}).

### Disable time-series storage

Expand Down
2 changes: 1 addition & 1 deletion src/current/v23.2/ui-overview-dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The **Overview** dashboard lets you monitor important SQL performance, replicati

To view this dashboard, [access the DB Console]({% link {{ page.version.version }}/ui-overview.md %}#db-console-access) and click **Metrics** on the left-hand navigation bar. The **Overview** dashboard is displayed by default.

The time-series data displayed in DB Console graphs is stored within the CockroachDB cluster and steadily increases for the first several days of a cluster's life, before an automatic job begins to prune it. By default, time-series data is stored for at 10-second resolution for 10 days, and at 30-minute resolution for 90 days. For details about managing this process, see this [How Can I Reduce or Disable the Storage of Time-series Data?]({% link {{ page.version.version }}/operational-faqs.md %}#can-i-reduce-or-disable-the-storage-of-time-series-data). In a new cluster, you will observe a steady increase in disk usage and the number of ranges even if you aren't writing data to the cluster.
The time-series data displayed in DB Console graphs is stored within the CockroachDB cluster and steadily increases for the first several days of a cluster's life, before an automatic job begins to prune it. By default, time-series data is stored for at 10-second resolution for 10 days, and at 30-minute resolution for 90 days. For details about managing this process, see [Can I reduce the storage of time-series data?]({% link {{ page.version.version }}/operational-faqs.md %}#can-i-reduce-the-storage-of-time-series-data). In a new cluster, you will observe a steady increase in disk usage and the number of ranges even if you aren't writing data to the cluster.

## Dashboard navigation

Expand Down
2 changes: 1 addition & 1 deletion src/current/v24.1/cluster-setup-troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,7 @@ Like any database system, if you run out of disk space the system will no longer
- [What happens when a node runs out of disk space?]({% link {{ page.version.version }}/operational-faqs.md %}#what-happens-when-a-node-runs-out-of-disk-space)
- [Why is memory usage increasing despite lack of traffic?]({% link {{ page.version.version }}/operational-faqs.md %}#why-is-memory-usage-increasing-despite-lack-of-traffic)
- [Why is disk usage increasing despite lack of writes?]({% link {{ page.version.version }}/operational-faqs.md %}#why-is-disk-usage-increasing-despite-lack-of-writes)
- [Can I reduce or disable the storage of timeseries data?]({% link {{ page.version.version }}/operational-faqs.md %}#can-i-reduce-or-disable-the-storage-of-time-series-data)
- [Can I reduce the storage of timeseries data?]({% link {{ page.version.version }}/operational-faqs.md %}#can-i-reduce-the-storage-of-time-series-data)

In rare cases, disk usage can increase on nodes with [Raft followers]({% link {{ page.version.version }}/architecture/replication-layer.md %}#raft) due to a [leader-leaseholder split]({% link {{ page.version.version }}/architecture/replication-layer.md %}#leader-leaseholder-splits).

Expand Down
6 changes: 0 additions & 6 deletions src/current/v24.1/kibana.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,12 +111,6 @@ Click **Refresh**. The query metrics will appear on the dashboard:

<img src="{{ 'images/v24.1/kibana-crdb-dashboard-sql.png' | relative_url }}" alt="CockroachDB Overview dashboard for Metricbeat with SQL metrics" style="border:1px solid #eee;max-width:100%" />

## Step 5. Disable DB Console's local storage of metrics (optional)

If you rely on external tools such as Kibana for storing and visualizing your cluster's time-series metrics, Cockroach Labs recommends that you [disable the DB Console's storage of time-series metrics]({% link {{ page.version.version }}/operational-faqs.md %}#disable-time-series-storage).

When storage of time-series metrics is disabled, the cluster continues to expose its metrics via the [Prometheus endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#prometheus-endpoint). The DB Console stops storing new time-series cluster metrics and eventually deletes historical data. The Metrics dashboards in the DB Console are still available, but their visualizations are blank. This is because the dashboards rely on data that is no longer available. You can create queries, visualizations, and alerts in Kibana based on the data it is collecting from your cluster's Prometheus endpoint.

## See also

- [Monitoring and Alerting]({% link {{ page.version.version }}/monitoring-and-alerting.md %})
Expand Down
6 changes: 0 additions & 6 deletions src/current/v24.1/monitor-cockroachdb-with-prometheus.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,12 +201,6 @@ Although Prometheus lets you graph metrics, [Grafana](https://grafana.com/) is a

1. [Add the dashboards to Grafana](http://docs.grafana.org/reference/export_import/#importing-a-dashboard).

## Step 6. Disable DB Console's local storage of metrics (optional)

If you rely on external tools such as Prometheus for storing and visualizing your cluster's time-series metrics, Cockroach Labs recommends that you [disable the DB Console's storage of time-series metrics]({% link {{ page.version.version }}/operational-faqs.md %}#disable-time-series-storage).

When storage of time-series metrics is disabled, the cluster continues to expose its metrics via the [Prometheus endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#prometheus-endpoint). The DB Console stops storing new time-series cluster metrics and eventually deletes historical data. The Metrics dashboards in the DB Console are still available, but their visualizations are blank. This is because the dashboards rely on data that is no longer available. You can create queries, visualizations, and alerts in Prometheus and AlertManager based on the data Prometheus is collecting from your cluster's Prometheus endpoint.

## See also

- [Monitoring and Alerting]({% link {{ page.version.version }}/monitoring-and-alerting.md %})
Expand Down
8 changes: 1 addition & 7 deletions src/current/v24.1/monitoring-and-alerting.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,7 @@ Each cluster automatically exposes its metrics at an [endpoint in Prometheus for
- Allow you to create and share dashboards, reports, and alerts based on metrics.
- Do not run within the cluster, and can help you to investigate a situation that led up to cluster outage even if the cluster is unavailable.

Metrics collected by the DB Console are stored within the cluster, and the SQL queries that create the reports on the Metrics dashboards also impose load on the cluster. To avoid this additional load, or if you rely on external tools for storing and visualizing your cluster's time-series metrics, Cockroach Labs recommends that you [disable the DB Console's storage of time-series metrics]({% link {{ page.version.version }}/operational-faqs.md %}#disable-time-series-storage).

When storage of time-series metrics is disabled, the cluster continues to expose its metrics via the [Prometheus endpoint](#prometheus-endpoint). The DB Console stops storing new time-series cluster metrics and eventually deletes historical data. The Metrics dashboards in the DB Console are still available, but their visualizations are blank. This is because the dashboards rely on data that is no longer available.
Metrics collected by the DB Console are stored within the cluster, and the SQL queries that create the reports on the Metrics dashboards also impose load on the cluster.

#### SQL Activity pages

Expand Down Expand Up @@ -1042,10 +1040,6 @@ Many of the [third-party monitoring integrations]({% link {{ page.version.versio

If you have configured [Prometheus]({% link {{ page.version.version }}/monitor-cockroachdb-with-prometheus.md %}) to monitor your CockroachDB instance, you can also configure alerting rule definitions to have Alertmanager detect [important events](#events-to-alert-on) and alert you when they occur.

If you rely on external tools for storing and visualizing your cluster's time-series metrics, Cockroach Labs recommends that you [disable the DB Console's storage of time-series metrics]({% link {{ page.version.version }}/operational-faqs.md %}#disable-time-series-storage).

When storage of time-series metrics is disabled, the DB Console Metrics dashboards in the DB Console are still available, but their visualizations are blank. This is because the dashboards rely on data that is no longer available.

#### Prometheus alerting rules endpoint

Every CockroachDB node exports an alerting rules template at `http://<host>:<http-port>/api/v2/rules/`. These rule definitions are formatted for easy integration with Alertmanager.
Expand Down
Loading
Loading