[asterisk-dev] [Code Review] A new higher-level API for working with Asterisk configs, with example code in app_skel.c and udptl.c
Matt Jordan
reviewboard at asterisk.org
Tue May 15 13:57:35 CDT 2012
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/1873/#review6216
-----------------------------------------------------------
A few global comments:
1. Very nifty.
2. This needs more doxygen comments.
3. Before this goes in, it needs comprehensive unit tests. All of the behaviour of the framework is testable using the unit test framework.
By far, the most difficult part of this framework - and probably the most 'easy' to get wrong aspect of this - is the relationship between what we are currently calling 'pvt' objects and 'pvt_config' objects. Part of that is because currently, for objects that would qualify as 'pvt' objects (sip_peer, ast_vm_user, etc.), those objects contain all of their configuration information in themselves. The idea that the configuration information lies in another object that has a different lifetime, e.g., during a reload that object is likely to be replaced, is tricky. People using the framework will have to be very careful recognizing the appropriate lifetime of their configuration object, such that operations occurring within that module get the appropriate configuration information throughout the entire lifetime of that operation, even if a reload happens.
I'm not sure there's any easier way to do this then what we have, but its something to keep in mind going forward. We should attempt to make the relationship between the 'pvt' and 'pvt_config' as intuitive as possible, and make it as difficult as possible to create relationships that result in using 'new' configuration information after reloads on operations that already under way (or worse, reference leaks).
/trunk/Makefile
<https://reviewboard.asterisk.org/r/1873/#comment11466>
No compiley on gcc 4.5.1. Need to check that the appropriate compiler version is in place before enabling this warning.
/trunk/apps/app_skel.c
<https://reviewboard.asterisk.org/r/1873/#comment11464>
While we're here, we may as well update the location of the coding guidelines to point towards the asterisk.org wiki (there isn't a CODING_GUIDELINES in my checkout of trunk at any rate)
/trunk/apps/app_skel.c
<https://reviewboard.asterisk.org/r/1873/#comment11457>
Even though this is a 'sample' application with dummy config objects, all of this needs doxygen comments (including the later functions, etc.). As it is, this app doesn't follow the coding guidelines it specifies in the header :-)
/trunk/apps/app_skel.c
<https://reviewboard.asterisk.org/r/1873/#comment11456>
This is probably one of the more confusing aspects of this change.
Its not clear just from the example application what the purpose of this structure is, or how its related to skel_pvt_config. I imagine the idea here is to show how a private in memory object (such as a sip_peer or an ast_vm_user) can be populated from the configuration information in a file; in that case, this needs to be spelled out explicitly. You may also want to expand upon the information in skel_pvt to show more explicitly its relationship to skel_pvt_config. A few things that should be demonstrated:
* How to manage the lifetime of a skel_pvt's configuration and/or its information over multiple method calls
* How a skel_pvt should look up its configuration information
Some of this is shown in the CLI command that iterates over the skel_pvt's, but given the concise nature of that example, its hard to understand all of the relationships between these two objects.
/trunk/apps/app_skel.c
<https://reviewboard.asterisk.org/r/1873/#comment11458>
It would be nice if this demonstrated how to specify multiple entries in a white/black list
/trunk/apps/app_skel.c
<https://reviewboard.asterisk.org/r/1873/#comment11465>
From a coding guideline perspective, the '0' here is something of a magic number. If we had multiple objects in our ao2 global container, you could end up with things like:
RAII_VAR(struct blah_config *, blah, ao2_global_obj_ref(globals, 3), ao2_cleanup);
What says '3' is where blah is stored?
If nothing else, that should be a #define:
RAII_VAR(struct blah_config *, blah, ao2_global_obj_ref(globals, BLAH_CONFIG_INDEX), ao2_cleanup);
/trunk/apps/app_skel.c
<https://reviewboard.asterisk.org/r/1873/#comment11454>
It doesn't appear as if this will link the created private back into the cfg->pvts container - and, unless I'm missing something, it should. Looking at what calls this, it doesn't appear as if it links the private back into the container as well (which it probably can't, since it doesn't know if what was returned was 'found' or 'created').
I'm not sure I like dual purpose methods like this. I'd prefer the underlying calls to either find the object, or, if that fails, create it and link it back into cfg->pvt itself. That takes the burden off of the application writer to remember to link a created private into the underlying config structure.
/trunk/apps/app_skel.c
<https://reviewboard.asterisk.org/r/1873/#comment11473>
This example needs to demonstrate the recommended mechanism of tying a private and its config object together, and how a private should access its config object during in-flight operations. The access of the configuration object for the current instance of the private object should not be affected by a reload.
/trunk/apps/app_skel.c
<https://reviewboard.asterisk.org/r/1873/#comment11475>
I do think it would be useful to have the framework allow for a callback function to be executed on a pvt when its related config object is updated. That way, during a configuration reload, only those objects whose configuration was changed would need to update themselves.
/trunk/apps/app_skel.c
<https://reviewboard.asterisk.org/r/1873/#comment11474>
This is very cool. That is all.
/trunk/configs/app_skel.conf.sample
<https://reviewboard.asterisk.org/r/1873/#comment11459>
In addition to having a pvt that contains no information, add a pvt that contains a subset of the allowed fields (such as only bit1 and bit2)
/trunk/include/asterisk/config_options.h
<https://reviewboard.asterisk.org/r/1873/#comment11469>
If this is for internal use only, you may want to create an opaque struct and hide the void *new_config on it.
/trunk/include/asterisk/config_options.h
<https://reviewboard.asterisk.org/r/1873/#comment11477>
Why not use an AST_LIST here instead?
/trunk/include/asterisk/config_options.h
<https://reviewboard.asterisk.org/r/1873/#comment11485>
How do we handle modules that use multiple configuration files, e.g., anything that uses users.conf?
/trunk/include/asterisk/config_options.h
<https://reviewboard.asterisk.org/r/1873/#comment11462>
I'd prefer an additional process config option as well, that would allow for passing in an in memory ast_config object. This would be particularly useful for unit testing, but could also be useful in things like app_voicemail, where meta data of voicemails is stored in ast_config objects and may potentially be manipulated prior to loading into in memory objects.
/trunk/include/asterisk/config_options.h
<https://reviewboard.asterisk.org/r/1873/#comment11460>
Is there a way to specify whether or not an option is required?
We may have situations where we require an option to be in a configuration file, and do not want to provide a default value (if that's a bogus use case, feel free to discard this comment)
/trunk/include/asterisk/config_options.h
<https://reviewboard.asterisk.org/r/1873/#comment11470>
Use \code and \endcode around code samples (applies to all comments in this header)
/trunk/include/asterisk/config_options.h
<https://reviewboard.asterisk.org/r/1873/#comment11461>
Blob!
/trunk/include/asterisk/utils.h
<https://reviewboard.asterisk.org/r/1873/#comment11471>
Put a comment on RAII_VAR describing its usage
/trunk/main/astobj2.c
<https://reviewboard.asterisk.org/r/1873/#comment11467>
This needs to complain loudly if this occurs. You may be safe in assigning NULL, but at the same time someone requested an object that either (a) doesn't exist in the global array, or (b) is past the bounds of the global array. Particularly in the latter case, that needs to warn the user that there's a potential misconfiguration of (or programmatic error in) the application.
/trunk/main/astobj2.c
<https://reviewboard.asterisk.org/r/1873/#comment11468>
Again, complain if this occurs. Not getting the read lock would imply a pretty nasty problem, and future operations could fail spectacularly.
/trunk/main/config_options.c
<https://reviewboard.asterisk.org/r/1873/#comment11481>
The internal private structures and methods preferably need some doxygen comments
/trunk/main/config_options.c
<https://reviewboard.asterisk.org/r/1873/#comment11479>
A size of 80 may not be sufficient to hold the error message returned from regerror.
The proper way to handle this would be to first call regerror with an errbuff_size of 0 to determine the size of the buffer needed, then call regerror again with that buffer size. While that complicates this code to some extent, it would be very useful to report the full error returned from regcomp.
/trunk/main/config_options.c
<https://reviewboard.asterisk.org/r/1873/#comment11476>
Complain loudly if this happens
/trunk/main/config_options.c
<https://reviewboard.asterisk.org/r/1873/#comment11478>
We may want to use an AST_LIST traversal here instead - without a SENTINEL, this will seg fault. Since ensuring that a SENTINEL is placed on the end of the list is in the domain of the user of the framework, the current implementation allows for more coding errors then the AST_LIST macros would allow.
Can we get into a situation where we don't get a match? If so, we should probably throw out a warning.
/trunk/main/config_options.c
<https://reviewboard.asterisk.org/r/1873/#comment11455>
This feels odd. Why would we need a 'find_or_create_pvt', if we already failed to find it? Shouldn't we just need a 'create_pvt' option?
/trunk/main/config_options.c
<https://reviewboard.asterisk.org/r/1873/#comment11480>
Mark with \internal
/trunk/main/udptl.c
<https://reviewboard.asterisk.org/r/1873/#comment11482>
If these are going to be used in multiple implementation files (which I imagine they are), we may as well #define them in utils.h and give it a prefix of AST_.
/trunk/main/udptl.c
<https://reviewboard.asterisk.org/r/1873/#comment11483>
I think I had this on the previous review as well. These need to also be in the udptl_global_options struct.
/trunk/main/udptl.c
<https://reviewboard.asterisk.org/r/1873/#comment11484>
Since we're in here, may as well properly document all of these structs.
- Matt
On May 15, 2012, 12:40 a.m., Terry Wilson wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/1873/
> -----------------------------------------------------------
>
> (Updated May 15, 2012, 12:40 a.m.)
>
>
> Review request for Asterisk Developers.
>
>
> Summary
> -------
>
> This review supersedes the ones at reviews 1840 and 1841. There is still a lot of cleanup/documentation work to do. This review is mostly about the overall idea/method. Doing the finishing work before someone comes up with why it is all a bad idea seems like...a bad idea. If you want to comment on non-big-picture stuff, feel free, but most important at this stage is whether or not this is a good idea at all.
>
> The goal of this patch is to come up with a way to safely handle the loading and reloading a config data. Data should be consistent at all times, even if there is invalid data in the config file. Current modules tend to store the config-related data in private structures and modify it in-place as the config file is parsed. If there is a problem, the data is left in an inconsistent state.
>
> This solution decouples config-related data from the non-config-related state held in the various private structures. It should atomically swap the global, private, and private config-related data. It also adds a higher-level API for registering the various configuration options that exist at module load, with default callback handlers for common types and the ability to create custom handlers for other types. If the higher-level API is used, a few callback functions are defined and for the most part, config loading and reloading is done with a single function call. If the high-level API is not sufficient, it can either be modified as time-goes on, or a module can use the lower-level config option API functions themselves, keeping to the same overall format of swapping out config objects, etc. for thread safe reloads.
>
> This patch also makes significant use of the RAII_VAR macro which uses the gcc "cleanup" attribute to make sure that ref counts are cleaned up on return, etc.
>
> There needs to be a lot more documentation, unit tests, etc. But I should probably hold off on doing any of that until people have had a chance to look at the basic idea, etc. There are some configs that won't work with the high-level API as-is. Anything that uses categories that have the same name (chan_mobile) would need an option added that allows that. Things like ConfBridge with options that are user-definable DTMF codes would need a "catch-all" or pattern-matching for option names. Both would be fairly easy to implement.
>
> Rationale
> --------
> Why not store config data directly in the privates?
> Because updating the data at the time of parsing can leave things in an indeterminate state on reload.
>
> What about just storing the config data directly in the privates, and creating new privates as you parse and swap out for the old one?
> Swapping out the entire private structure would lose any non-config-related state in the private structure.
>
> What about using a copy function for the private's non-config-related state?
> Having to define(and keep it updated as new fields are added) a copy function for every private structure (and essentially for every type stored in that structure) that needs to properly handle reloads sounds like a huge pain to me.
>
> What about instead of having separate containers for privates and private configs, you just store a pointer to the private config in the private structure itself?
> There are two problems I see with this. 1) To ensure data is consistent when accessing multiple fields, one would need to hold a reference to the cfg in the private. But, since it is just a pointer, it encourages people to use it directly without grabbing a reference. By separating the containers, one must look up the config object and get a reference to it to be able to use it. 2) If there is a problem in the middle of switching out the cfg pointers, you end up with some privates with new configs and some with old.
>
> Overview of how it works: You basically have the global aco_info struct that defines information pertaining to the whole config. Then there are aco_types which define category-level things like regex for what categories are supported for the type, allocation/lookup functions, whether it is for a single global object, or objects in containers, etc. Below that are aco_options, which define the options available for a given type. For example:
>
> struct aco_info cfg_info = {
> .module = AST_MODULE,
> .filename = "app_skel.conf"
> .apply_config = skel_apply_config,
> .preload = {"general", SENTINEL }, /* If you need to load some contexts in order */
> };
>
> struct skel_global_cfg {
> ...
> };
>
> struct skel_pvt_cfg {
> ...
> };
>
> struct skel_pvt {
> ...
> };
>
> enum {
> GLOBAL_OPTIONS = 0,
> PVT_CFG_CONTAINER,
> PVT_CONTAINER,
> /* Must be declared last */
> NUM_GLOBAL_OBJECTS,
> };
> static AO2_GLOBAL_OBJ_STATIC(global_config, NUM_GLOBAL_OBJECTS);
> AST_MUTEX_DEFINE_STATIC(reload_lock);
>
> /* Required for global */
> void *skel_global_cfg_alloc(const char*cat);
>
> /* Required for privates (container-stored objects) */
> void *skel_pvt_cfg_alloc(const char *cat);
> void *skel_pvt_find_or_create(const char *cat);
> void *skel_pvt_find_in_container(struct ao2_container *cont, const char *cat);
> int skel_pvt_containers_alloc(struct ao2_container **newpvts, struct ao2_container **newcfgs);
>
> /* Optional for privates */
> int skel_pvt_cfg_post_init(void *cfg); /* Could be used to inherit global settings...ew. */
> int skel_pvt_cfg_pre_link(void *cfg); /* Could be used for final verification that things look a-ok */
>
> static int apply_config(void)
> {
> RAII_VAR(void *, new_global, aco_info_new_global_get(&cfg_info, "global"), ao2_cleanup);
> RAII_VAR(struct ao2_container *, new_pvts, aco_info_new_privates_get(&cfg_info, "private"), ao2_cleanup);
> RAII_VAR(struct ao2_container *, new_cfgs, aco_info_new_configs_get(&cfg_info, "private"), ao2_cleanup);
>
> if (!(new_global && new_pvts && new_cfgs)) {
> return -1;
> }
> /* Do any fixup for global configs here, individual privates could be fixed up via the pre-link callback */
>
> ao2_global_obj_replace_unref(global_config, GLOBAL_OPTIONS, new_global);
> ao2_global_obj_replace_unref(global_config, PVT_CONTAINER, new_pvts);
> ao2_global_obj_replace_unref(global_config, PVT_CFG_CONTAINER, new_cfgs);
>
> return 0;
> }
>
> static int process_config(int reload)
> {
> ast_mutex_lock(&reload_lock);
> if (aco_process_config(&cfg_info, reload)) {...};
> ast_mutex_unlock(&reload_lock);
> ...
> }
>
> static int reload(void)
> {
> if (process_config(1)) {...}
> }
> static int load_module(void)
> {
> ...
> aco_info_init(&cfg_info));
> global_type = aco_type_global_alloc("global", CONTEXT_ALLOW, "general", (aco_type_alloc) skel_global_alloc);
> priv_type = aco_type_private_alloc("private", CONTEXT_DENY, "general", NULL, NULL, (aco_type_alloc) skel_pvt_cfg_alloc, skel_containers_alloc, skel_find_or_create_pvt, skel_find_pvt, NULL, NULL)
>
> aco_type_register(&cfg_info, global_type);
> aco_type_register(&cfg_info, priv_type);
>
> aco_option_register(&cfg_info, "foo", global_type, "booya", OPT_STRINGFIELD_T, 0, STRFLDSET(struct skel_global_config, foo));
> ...
> if (process_config(0)) {...}
> ...
> }
>
>
> Diffs
> -----
>
> /trunk/Makefile 366506
> /trunk/apps/app_skel.c 366506
> /trunk/configs/app_skel.conf.sample PRE-CREATION
> /trunk/include/asterisk/astobj2.h 366506
> /trunk/include/asterisk/config.h 366506
> /trunk/include/asterisk/config_options.h PRE-CREATION
> /trunk/include/asterisk/stringfields.h 366506
> /trunk/include/asterisk/utils.h 366506
> /trunk/main/asterisk.exports.in 366506
> /trunk/main/astobj2.c 366506
> /trunk/main/config.c 366506
> /trunk/main/config_options.c PRE-CREATION
> /trunk/main/udptl.c 366506
>
> Diff: https://reviewboard.asterisk.org/r/1873/diff
>
>
> Testing
> -------
>
> Lots of testing with malloc debug, valgrind, etc.
>
>
> Thanks,
>
> Terry
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20120515/1201f7fe/attachment-0001.htm>
More information about the asterisk-dev
mailing list