<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 solid;">
<tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://reviewboard.asterisk.org/r/1873/">https://reviewboard.asterisk.org/r/1873/</a>
</td>
</tr>
</table>
<br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<p style="margin-top: 0;">On April 26th, 2012, 7 p.m., <b>rmudgett</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; border-collapse: collapse; margin: 2px padding: 2px;">
<thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; font-size: 9pt; padding: 4px 8px; text-align: left;">
<a href="https://reviewboard.asterisk.org/r/1873/diff/1/?file=27371#file27371line463" style="color: black; font-weight: bold; text-decoration: underline;">/main/config_options.c</a>
<span style="font-weight: normal;">
(Diff revision 1)
</span>
</th>
</tr>
</thead>
<tbody style="background-color: #e4d9cb; padding: 4px 8px; text-align: center;">
<tr>
<td colspan="2"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<td colspan="2"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">int aco_process_category_options(struct ao2_container *container, struct ast_config *cfg, const char *cat, void *obj)</pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">463</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">opt</span><span class="p">)</span> <span class="p">{</span></pre></td>
</tr>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">464</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">ast_log</span><span class="p">(</span><span class="n">LOG_WARNING</span><span class="p">,</span> <span class="s">"Could not find option suitable for category '%s' named '%s' at line %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">cat</span><span class="p">,</span> <span class="n">var</span><span class="o">-></span><span class="n">name</span><span class="p">,</span> <span class="n">var</span><span class="o">-></span><span class="n">lineno</span><span class="p">);</span></pre></td>
</tr>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">465</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="k">return</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span></pre></td>
</tr>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">466</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="p">}</span></pre></td>
</tr>
</tbody>
</table>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">You should not quit. Just give warning and ignore option. The option is either misspelled or not supported in this Asterisk version.
Also you can indicate the file name var->file in the message.
How about:
Unknown option '%s' in category '%s' at line %d of file %s.</pre>
</blockquote>
<p>On April 27th, 2012, 2:13 p.m., <b>Terry Wilson</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">No. I specifically want to quit. If someone edits and screws up their config, they should fix it. We shouldn't just warn and then have screwed up data. Tell them it is broken, and don't do the reload. There is absolutely no good reason to do a reload when we *know* that the data is bad.</pre>
</blockquote>
<p>On April 27th, 2012, 5:12 p.m., <b>rmudgett</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">I was thinking about specifying an option that is not available in the running version but is available in another version. Switching between versions happens a lot and is a bit of a nuisance to also update config files when the unknown option can be safely ignored.
It is why you are using a LOG_WARNING instead of a LOG_ERROR right?</pre>
</blockquote>
<p>On April 27th, 2012, 5:19 p.m., <b>Terry Wilson</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">We generally do not remove options between versions anymore. Things get deprecated, but not often removed. And even then, people need to fix their stuff instead of having things randomly kind of work. One of the main reasons I designed things for this was to ensure that reloads always result in consistant data. I will not change this unless absolutely forced.</pre>
</blockquote>
<p>On April 27th, 2012, 5:42 p.m., <b>Tilghman Lesher</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">No, but people do revert to earlier branches. Consider some point down the road that somebody tries a particular beta of Asterisk 12, with some new options, and then reverts back to Asterisk 11. If they've specified new options that are specific to Asterisk 12, then this code would consider those options "errors" and refuse to load/reload the configuration. That's bad.
And furthermore, your use of WARNING in this situation contravenes our standards for what is a warning and what is an error. An error is unrecoverable, so we quit. A warning specifies that something should probably be changed, but we will likely be able to correct for the problem and do the right thing anyway. If you quit, it should be an error. If you continue, it should be a warning.</pre>
</blockquote>
<p>On April 27th, 2012, 5:48 p.m., <b>rmudgett</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Things get "removed" when you switch to an earlier version that does not have the new feature. It is up to the user to fix the configs. An unknown option should not block loading the config. It should generate a warning that the user should fix *if* he so desires.</pre>
</blockquote>
<p>On April 27th, 2012, 6:13 p.m., <b>Terry Wilson</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Doesn't matter. Not changing. :-) kpfleming had a good example. A user mistypes secret as sceret. Is it a good idea to go ahead and reload things when we know that doesn't mean anything? We do the reload, and all of the sudden the peer has no secret and is completely insecure. Get used to it. It isn't changing. :-) No amount of small inconvenience when doing something that very infrequently happens is worth allowing inconsistant data to be loaded in the system.
Also, I have already converted the WARNINGS to ERRORS to make things more clear.</pre>
</blockquote>
</blockquote>
<pre style="margin-left: 1em; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">We should probably take this to the asterisk-dev mailing list for easier communication, but I completely agree that his has been a long-standing failing of Asterisk on the whole. If a config file (or Realtime table) contains data that the module cannot process (misnamed options, invalid values, syntax errors, etc.) that entire configuration content should be ignored for safety. The overwhelming majority of such occurrences are mistakes made by the person configuring Asterisk, not options left over from trying a new/different branch. In those rare cases where we do in fact deprecate/remove a configuration option, the code can still allow the config parser to accept it, but ignore its content and generate a warning about the fact that it is being ignored.</pre>
<br />
<p>- Kevin</p>
<br />
<p>On April 16th, 2012, 1:45 p.m., Terry Wilson wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" style="background-image: url('https://reviewboard.asterisk.org/media/rb/images/review_request_box_top_bg.png'); background-position: left top; background-repeat: repeat-x; border: 1px black solid;">
<tr>
<td>
<div>Review request for Asterisk Developers.</div>
<div>By Terry Wilson.</div>
<p style="color: grey;"><i>Updated April 16, 2012, 1:45 p.m.</i></p>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">This review supersedes the ones at reviews 1840 and 1841. There is still a lot of cleanup/documentation work to do. This review is mostly about the overall idea/method. Doing the finishing work before someone comes up with why it is all a bad idea seems like...a bad idea. If you want to comment on non-big-picture stuff, feel free, but most important at this stage is whether or not this is a good idea at all.
The goal of this patch is to come up with a way to safely handle the loading and reloading a config data. Data should be consistent at all times, even if there is invalid data in the config file. Current modules tend to store the config-related data in private structures and modify it in-place as the config file is parsed. If there is a problem, the data is left in an inconsistent state.
This solution decouples config-related data from the non-config-related state held in the various private structures. It should atomically swap the global, private, and private config-related data. It also adds a higher-level API for registering the various configuration options that exist at module load, with default callback handlers for common types and the ability to create custom handlers for other types. If the higher-level API is used, a few callback functions are defined and for the most part, config loading and reloading is done with a single function call. If the high-level API is not sufficient, it can either be modified as time-goes on, or a module can use the lower-level config option API functions themselves, keeping to the same overall format of swapping out config objects, etc. for thread safe reloads.
This patch also makes significant use of the RAII_VAR macro which uses the gcc "cleanup" attribute to make sure that ref counts are cleaned up on return, etc.
There needs to be a lot more documentation, unit tests, etc. But I should probably hold off on doing any of that until people have had a chance to look at the basic idea, etc. There are some configs that won't work with the high-level API as-is. Anything that uses categories that have the same name (chan_mobile) would need an option added that allows that. Things like ConfBridge with options that are user-definable DTMF codes would need a "catch-all" or pattern-matching for option names. Both would be fairly easy to implement.
Rationale
--------
Why not store config data directly in the privates?
Because updating the data at the time of parsing can leave things in an indeterminate state on reload.
What about just storing the config data directly in the privates, and creating new privates as you parse and swap out for the old one?
Swapping out the entire private structure would lose any non-config-related state in the private structure.
What about using a copy function for the private's non-config-related state?
Having to define(and keep it updated as new fields are added) a copy function for every private structure (and essentially for every type stored in that structure) that needs to properly handle reloads sounds like a huge pain to me.
What about instead of having separate containers for privates and private configs, you just store a pointer to the private config in the private structure itself?
There are two problems I see with this. 1) To ensure data is consistent when accessing multiple fields, one would need to hold a reference to the cfg in the private. But, since it is just a pointer, it encourages people to use it directly without grabbing a reference. By separating the containers, one must look up the config object and get a reference to it to be able to use it. 2) If there is a problem in the middle of switching out the cfg pointers, you end up with some privates with new configs and some with old.
Overview of how it works: You basically have the global aco_info struct that defines information pertaining to the whole config. Then there are aco_types which define category-level things like regex for what categories are supported for the type, allocation/lookup functions, whether it is for a single global object, or objects in containers, etc. Below that are aco_options, which define the options available for a given type. For example:
struct aco_info cfg_info = {
.module = AST_MODULE,
.filename = "app_skel.conf"
.apply_config = skel_apply_config,
.preload = {"general", SENTINEL }, /* If you need to load some contexts in order */
};
struct skel_global_cfg {
...
};
struct skel_pvt_cfg {
...
};
struct skel_pvt {
...
};
enum {
GLOBAL_OPTIONS = 0,
PVT_CFG_CONTAINER,
PVT_CONTAINER,
/* Must be declared last */
NUM_GLOBAL_OBJECTS,
};
static AO2_GLOBAL_OBJ_STATIC(global_config, NUM_GLOBAL_OBJECTS);
AST_MUTEX_DEFINE_STATIC(reload_lock);
/* Required for global */
void *skel_global_cfg_alloc(const char*cat);
/* Required for privates (container-stored objects) */
void *skel_pvt_cfg_alloc(const char *cat);
void *skel_pvt_find_or_create(const char *cat);
void *skel_pvt_find_in_container(struct ao2_container *cont, const char *cat);
int skel_pvt_containers_alloc(struct ao2_container **newpvts, struct ao2_container **newcfgs);
/* Optional for privates */
int skel_pvt_cfg_post_init(void *cfg); /* Could be used to inherit global settings...ew. */
int skel_pvt_cfg_pre_link(void *cfg); /* Could be used for final verification that things look a-ok */
static int apply_config(void)
{
RAII_VAR(void *, new_global, aco_info_new_global_get(&cfg_info, "global"), ao2_cleanup);
RAII_VAR(struct ao2_container *, new_pvts, aco_info_new_privates_get(&cfg_info, "private"), ao2_cleanup);
RAII_VAR(struct ao2_container *, new_cfgs, aco_info_new_configs_get(&cfg_info, "private"), ao2_cleanup);
if (!(new_global && new_pvts && new_cfgs)) {
return -1;
}
/* Do any fixup for global configs here, individual privates could be fixed up via the pre-link callback */
ao2_global_obj_replace_unref(global_config, GLOBAL_OPTIONS, new_global);
ao2_global_obj_replace_unref(global_config, PVT_CONTAINER, new_pvts);
ao2_global_obj_replace_unref(global_config, PVT_CFG_CONTAINER, new_cfgs);
return 0;
}
static int process_config(int reload)
{
ast_mutex_lock(&reload_lock);
if (aco_process_config(&cfg_info, reload)) {...};
ast_mutex_unlock(&reload_lock);
...
}
static int reload(void)
{
if (process_config(1)) {...}
}
static int load_module(void)
{
...
aco_info_init(&cfg_info));
global_type = aco_type_global_alloc("global", CONTEXT_ALLOW, "general", (aco_type_alloc) skel_global_alloc);
priv_type = aco_type_private_alloc("private", CONTEXT_DENY, "general", NULL, NULL, (aco_type_alloc) skel_pvt_cfg_alloc, skel_containers_alloc, skel_find_or_create_pvt, skel_find_pvt, NULL, NULL)
aco_type_register(&cfg_info, global_type);
aco_type_register(&cfg_info, priv_type);
aco_option_register(&cfg_info, "foo", global_type, "booya", OPT_STRINGFIELD_T, 0, STRFLDSET(struct skel_global_config, foo));
...
if (process_config(0)) {...}
...
}</pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Lots of testing with malloc debug, valgrind, etc.</pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>/trunk/main/config_options.c <span style="color: grey">(PRE-CREATION)</span></li>
<li>/trunk/main/config.c <span style="color: grey">(362149)</span></li>
<li>/trunk/main/asterisk.exports.in <span style="color: grey">(362149)</span></li>
<li>/trunk/main/astobj2.c <span style="color: grey">(362149)</span></li>
<li>/trunk/include/asterisk/utils.h <span style="color: grey">(362149)</span></li>
<li>/trunk/include/asterisk/stringfields.h <span style="color: grey">(362149)</span></li>
<li>/trunk/include/asterisk/config_options.h <span style="color: grey">(PRE-CREATION)</span></li>
<li>/trunk/include/asterisk/config.h <span style="color: grey">(362149)</span></li>
<li>/trunk/include/asterisk/astobj2.h <span style="color: grey">(362149)</span></li>
<li>/trunk/configs/app_skel.conf.sample <span style="color: grey">(PRE-CREATION)</span></li>
<li>/trunk/apps/app_skel.c <span style="color: grey">(362149)</span></li>
<li>/trunk/main/udptl.c <span style="color: grey">(362149)</span></li>
</ul>
<p><a href="https://reviewboard.asterisk.org/r/1873/diff/" style="margin-left: 3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>