[asterisk-dev] How to improve astdb? Any ideas?

[Stefan Schmidt]

Hello list,

in my test scenario i have seen that astdb could be a problem in a heavy
loaded system (for example 2000 sip peers). Atleast the real problem is
the astdb->sync which is called on every astdb_put/delete or deltree call.

the problem is that syncing the astdb to the disk will needs minimum 2ms
and up to 40 ms with an big amount of entries in the database. In this
time no incoming package is read so this will cause some really bad
things to the system like every peer get unreachable.

i have thought about 3 different solutions to make astdb perform better.
this are just 3 ways which came to my mind, maybe there are much better
ideas or solutions for this so please tell me.

1.)
using a global counter and a scheduler to sync the astdb only on every
100. change, or atleast every (or 5) second if there has been a change.

this would cause a possible data loss of 1 second if the system crash.
for example if there are 3600 sip peers atleast one registration will be
lost and also other data which are written in the last second.
another problem of this solution would be the loss of data when astdb is
used from the dialplan. so maybe there should be a difference between
sip data and dialplan data.

this solution will decrease system load but the highest amount of data
loss is possible.

2.)
creating an own thread which have its own astdb. if there is a change
the current astdb is copied to the astdb of this thread and then synced.
if there is another change meanwhile the thread would be looked and
pendingflag would be set, to sync immediatly after the last sync is
finished.

this would cause a possible data loss of several milliseconds, depending
how long an astdb-> sync will take. The chance of loosing data is much
smaller cause the amount of unsync data would be smaller. The problem is
that there still would be a high system load cause of the disk io (same
as now).
and i dont know how to duplicate an astdb or even if its possible to
have two astdb structures in use. One to sync and another to be init on
startup.

3.)
is nearly the same like 2. but changing the astdb which saves the
entries, and remove it with another method of data storage. i think of
using a link list, a tree structure or a container or something else.
like above if something changed the second threat imports the changes to
one astdb structure and sync it to disk.

even this solution is much more difficult is see a big improvment with
this. if we split up the functions which put, get and del data from the
astdb into a frontent and a backend, its much easier to build an
interface for using astdb with the realtime API


i hope there would be some ideas which way this improvment should be
gone. i am not sure what would be the best.

thanks!

best regards

steve