Exception in thread 'OrientDB WAL Flush Task'

[quanghaisoft]

OrientDB Version: 3.0.31

Java Version: java version "1.8.0_211"

OS: ubuntu

Expected behavior

Upsert success, not raise any error

Actual behavior

Exception in thread 'OrientDB WAL Flush Task'
java.nio.BufferOverflowException
	at java.nio.Buffer.nextPutIndex(Buffer.java:527)
	at java.nio.DirectByteBuffer.putInt(DirectByteBuffer.java:706)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.cas.OCASDiskWriteAheadLog$RecordsWriter.run(OCASDiskWriteAheadLog.java:1920)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Can not update/insert data any more

Steps to reproduce

My OrientDB system has over 100 classes with over 50GB data, setup in cluster mode with 6 node, on 6 different IP.
When I upsert big data into 1 master node orientdb in distributed ,
Error appear in logs

Exception in thread 'OrientDB WAL Flush Task'
java.nio.BufferOverflowException
	at java.nio.Buffer.nextPutIndex(Buffer.java:527)
	at java.nio.DirectByteBuffer.putInt(DirectByteBuffer.java:706)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.cas.OCASDiskWriteAheadLog$RecordsWriter.run(OCASDiskWriteAheadLog.java:1920)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[andrii0lomakin]

Hi, @quanghaisoft what is the WAL page size in your environment? You can find size of the WAL page in the log. Do you set any non-default settings? Thank you for the bug report.

[quanghaisoft]

thanks you for your answer , this is logs, recent day, this error not appear
config in server.sh
-Dstorage.wal.maxSegmentSize=128 -Dstorage.wal.maxSize=16384

2020-06-03 17:01:01:421 INFO Allocation of 972800 pages. [OEngineLocalPaginated]
2020-06-03 17:01:03:978 INFO WAL maximum segment size is set to 6,144 MB [OrientDBDistributed]
2020-06-03 17:01:03:978 INFO Databases directory: /data/orientdb/databases [OServer]
2020-06-03 17:01:04:039 INFO Direct IO for WAL located in /data/orientdb/databases/OSystem is allowed with block size 4096 bytes. [OCASDiskWriteAheadLog]
2020-06-03 17:01:04:040 INFO Page size for WAL located in /data/orientdb/databases/OSystem is set to 4096 bytes. [OCASDiskWriteAheadLog]
2020-06-03 17:01:04:104 WARNI Storage 'OSystem' was not closed properly. Will try to recover from write ahead log [OLocalPaginatedStorage]

[quanghaisoft]

We have written a lot of data simultaneously

[andrii0lomakin]

Hi @quanghaisoft . I am not sure that I understood you. Do you mean that now you can not reproduce this error on your tests?

[quanghaisoft]

This error does not show again. We are running in production, so I cannot test it. If it appears I will open this issue

[quanghaisoft]

this bug in my production so we copy data and rebuild cluster immediately.if it appears again,we will send it to you

[andrii0lomakin]

I see thank you, I will try to reproduce with settings which you sent to me.

[andrii0lomakin]

Hi @quanghaisoft what amount of RAM do you have on the server ?

[quanghaisoft]

we using more than 150G ram , CPU 24 core on node , with 6 node in cluster

[quanghaisoft]

when we using multi write upsert to one vertex in 3 master node, it slow down , can you help me a recommend Config to active fast upsert with bigdata, avoid lock record when confict happen,avoid not response .etc
this is current config: