[asterisk-dev] [Code Review]: Rework the Asterisk Test Suite's version handling

opticron reviewboard at asterisk.org
Fri Jun 15 14:49:10 CDT 2012 


> On June 15, 2012, 10:07 a.m., opticron wrote:
> > /asterisk/trunk/lib/python/asterisk/version.py, line 226
> > <https://reviewboard.asterisk.org/r/1982/diff/1/?file=28659#file28659line226>
> >
> >     This won't necessarily be true once Asterisk 11 is branched since trunk may contain new digiumphones related features for which tests have been written that can not be merged into a released branch.  In that case, 11-digiumphones would be the appropriate version to test against.
> 
> Matt Jordan wrote:
>     Not really.  It gets worse.
>     
>     When 11-digiumphones occurs, you will have tests that have the 'digiumphones' feature, with a minimum version of 11.x.y, that should not execute against a checkout of the 11 branch.  Unfortunately, you'll also have tests that have the 'digiumphones' feature, with a minimum version of 10, that should execute against a checkout of the 11 branch.  Thus, not only is the concept of a 'feature' not sufficient (at least so long as they both have the same feature name), but the concept that a branch version is greater then a tag version could also be suspect.
>     
>     That breaks the current logic in ways that I don't think we can justifiably handle
>     
>      We can either:
>     1) Come up with a new '12' feature name for stuff that gets added.  New tests would get a new feature name, and thus 'digiumphones' would continue to work as it does.
>     2) Branch the Test Suite.
>     
>     At this time, I'm thinking we will branch the Test Suite.  It'll be far easier to write tests for a particular branch (or trunk) and merge them then it is to continue with the version madness.

Right, that's the concept I was trying to get across (and obviously failed at).  Both options introduce a maintenance hurdle, but I think branching the testsuite is the right way to get this done.


- opticron


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/1982/#review6481
-----------------------------------------------------------


On June 8, 2012, 3:56 p.m., Matt Jordan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/1982/
> -----------------------------------------------------------
> 
> (Updated June 8, 2012, 3:56 p.m.)
> 
> 
> Review request for Asterisk Developers, Russell Bryant, Paul Belanger, and wdoekes.
> 
> 
> Summary
> -------
> 
> This patch updates the Asterisk Test Suite's handling of Asterisk versions.  Specifically, it does the following:
> 
> * Removes support for Asterisk versions 1.4 and 1.6.2.  While this is some what controversial, it makes version parsing slightly saner, as it doesn't have to handle the 1.6.x scheme of versions.  (1.4 was just a freebie in the removal)
> * Changes the internal representation of versions to essentially ignore the 'concept' number in the comparisons.  This is, after all, one of the reasons why we dropped the '1' in '1.10' in the first place - it adds no meaningful value.
> * Adds the concepts of 'features' to a version.  Features are not used in terms of comparisons, but instead are explicitly checked by things that want to determine if a particular test can run.  They are, however, parsed out of the version tag - so 10.6.0-digiumphones-rc1 is understood as being the first release candidate of 10.6.0 with a feature of digiumphones; 1.8.11-cert3 is understood as being the third release of 1.8.11 with a feature of cert, etc.
> * Completely re-done version string parsing.  I'm not sure if I made it uniquely better - our versions are pretty tough to parse in a sane fashion, what with team branch names, multiple numbering schemes, features, modifiers, and other iteration numbers sprinkled liberally throughout - but I gave it a shot.  It passes the unit tests (which were tweaked up slightly as well to test some things that I'm not sure we tested previously), and seems to work well.
> * Updated the minversion in all test-config.yaml to be a tag.  A minversion in a test is a tag.  It is not a branch.  (Unless you actually put branch-10, but that would be silly because then it would only ever be run for a checkout of a branch, and not for any tags - and why would that make sense?)  While this would still treat a minversion: 1.8 as a minimum version of tag 1.8.0.0, that nomenclature is unclear.  This updates it to be consistent and clear as to what a minimum version is.
> * Added a new option to test-config.yaml 'feature'.  If a test can only be run against digiumphones, you add 'digiumphones' as the feature.  If it can only be run against 'cert',  you add 'cert'.  If its run for both, just put 'cert', since that implies 'digiumphones'.  And there's an override (for now) to make Asterisk 11 run everything.
> 
> 
> This addresses bug ASTERISK-19818.
>     https://issues.asterisk.org/jira/browse/ASTERISK-19818
> 
> 
> Diffs
> -----
> 
>   /asterisk/trunk/README.txt 3259 
>   /asterisk/trunk/lib/python/asterisk/TestConfig.py 3259 
>   /asterisk/trunk/lib/python/asterisk/version.py 3259 
>   /asterisk/trunk/runtests.py 3259 
>   /asterisk/trunk/tests/agi/exit_status/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/bridge/bridge_baseline/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/bridge/bridge_transfer_callee/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/confbridge/confbridge_nominal/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/confbridge/confbridge_recording/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/directory_attendant_exit/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/directory_context_operator_exit/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/directory_operator_exit/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/incomplete/sip_incomplete/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/page/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/authenticate_extensions/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/authenticate_invalid_mailbox/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/authenticate_invalid_password/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/authenticate_nominal/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_callback/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_delete/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_dialout/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_envelope/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_forward/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_forward_hangup/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_forward_with_prepend/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_new_user/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_new_user_hangup/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_nominal/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_options_change_password/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_options_record_busy/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_options_record_name/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_options_record_temp/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_options_record_unavail/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_reply/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_while_leaving_msg/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/func_vmcount/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/leave_voicemail_external_notification/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/leave_voicemail_nominal/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/leave_voicemail_priority/test-config.yaml 3259 
>   /asterisk/trunk/tests/apps/voicemail/play_message/test-config.yaml 3259 
>   /asterisk/trunk/tests/blind-transfer-parkingtimeout/test-config.yaml 3259 
>   /asterisk/trunk/tests/callparking/test-config.yaml 3259 
>   /asterisk/trunk/tests/callparking_retrieval/test-config.yaml 3259 
>   /asterisk/trunk/tests/cause_answered_elsewhere/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/app_dial_G_flag/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/app_queue/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/blind-transfer-accountcode/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/cdr_accountcode/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/cdr_unanswered_yes/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/cdr_userfield/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/console_dial_sip_answer/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/console_dial_sip_busy/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/console_dial_sip_congestion/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/console_dial_sip_transfer/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/console_fork_after_busy_forward/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/console_fork_before_dial/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/nocdr/test-config.yaml 3259 
>   /asterisk/trunk/tests/cdr/originate-cdr-disposition/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/custom_info/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/generic_ccss/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/handle_response_address_incomplete/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/handle_response_refer/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/hangupcause/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/info_dtmf/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/invite_no_totag/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/message_auth/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/message_auth_cust_hdr/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/message_disabled/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/message_from_call/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/message_mark_all_outbound/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/message_send_ami/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/message_unauth/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/message_unauth_from/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/nat_supertest/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/noload_res_srtp/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/noload_res_srtp_attempt_srtp/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/options/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/pcap_demo/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/realtime_nosipregs/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/realtime_sipregs/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/refer_replaces_to_self/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/secure_bridge_media/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_attended_transfer/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_attended_transfer_tcp/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_attended_transfer_v6/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_cause/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_channel_params/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_custom_presence/multiple_state_change/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_custom_presence/nominal_state_change/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_custom_presence/non_digium_state_change/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_custom_presence/resubscribe/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_one_legged_transfer/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_one_legged_transfer_v6/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_outbound_address/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_register/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/sip_srtp/test-config.yaml 3259 
>   /asterisk/trunk/tests/channels/SIP/use_contact_from_200/test-config.yaml 3259 
>   /asterisk/trunk/tests/chanspy/chanspy_barge/test-config.yaml 3259 
>   /asterisk/trunk/tests/chanspy/chanspy_w_mixmonitor/test-config.yaml 3259 
>   /asterisk/trunk/tests/connected_line/macro/test-config.yaml 3259 
>   /asterisk/trunk/tests/connected_line/subroutine/test-config.yaml 3259 
>   /asterisk/trunk/tests/dialplan/test-config.yaml 3259 
>   /asterisk/trunk/tests/directed_pickup/test-config.yaml 3259 
>   /asterisk/trunk/tests/dynamic-modules/test-config.yaml 3259 
>   /asterisk/trunk/tests/example/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/channel-status/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/connect/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/control-stream-file/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/database/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/execute/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/get-data/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/hangup/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/record-file/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/say-alpha/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/say-date/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/say-datetime/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/say-digits/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/say-number/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/say-phonetic/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/say-time/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/stream-file/test-config.yaml 3259 
>   /asterisk/trunk/tests/fastagi/wait-for-digit/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_g711_t38/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_mix1/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_mix2/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_mix3/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_mix4/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_native_t38/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_native_t38_ced/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_no_t38/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_t38_g711/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_timeout1/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_timeout2/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_timeout3/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_timeout4/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/gateway_timeout5/test-config.yaml 3259 
>   /asterisk/trunk/tests/fax/local_channel_t38_queryoption/test-config.yaml 3259 
>   /asterisk/trunk/tests/feature_attended_transfer/test-config.yaml 3259 
>   /asterisk/trunk/tests/func_presencestate/test-config.yaml 3259 
>   /asterisk/trunk/tests/func_srv/test-config.yaml 3259 
>   /asterisk/trunk/tests/iax2/basic-call/test-config.yaml 3259 
>   /asterisk/trunk/tests/manager/action-events-response/test-config.yaml 3259 
>   /asterisk/trunk/tests/manager/authlimit/test-config.yaml 3259 
>   /asterisk/trunk/tests/manager/authtimeout/test-config.yaml 3259 
>   /asterisk/trunk/tests/manager/login/test-config.yaml 3259 
>   /asterisk/trunk/tests/manager/response-time/test-config.yaml 3259 
>   /asterisk/trunk/tests/masquerade/test-config.yaml 3260 
>   /asterisk/trunk/tests/mixmonitor/test-config.yaml 3259 
>   /asterisk/trunk/tests/mixmonitor_audiohook_inherit/test-config.yaml 3259 
>   /asterisk/trunk/tests/one-step-parking/test-config.yaml 3259 
>   /asterisk/trunk/tests/pbx/call-files/test-config.yaml 3259 
>   /asterisk/trunk/tests/pbx/call-files2/test-config.yaml 3259 
>   /asterisk/trunk/tests/pbx/merge_contexts/test-config.yaml 3259 
>   /asterisk/trunk/tests/pbx/pbx_lua_background/test-config.yaml 3259 
>   /asterisk/trunk/tests/pbx/pbx_lua_goto/test-config.yaml 3259 
>   /asterisk/trunk/tests/queues/position_priority_maxlen/test-config.yaml 3259 
>   /asterisk/trunk/tests/queues/queue_baseline/test-config.yaml 3259 
>   /asterisk/trunk/tests/queues/queue_transfer_callee/test-config.yaml UNKNOWN 
>   /asterisk/trunk/tests/queues/ringinuse_and_pause/test-config.yaml 3259 
>   /asterisk/trunk/tests/queues/set_penalty/test-config.yaml 3259 
>   /asterisk/trunk/tests/redirecting/macro/test-config.yaml 3259 
>   /asterisk/trunk/tests/redirecting/subroutine/test-config.yaml 3259 
>   /asterisk/trunk/tests/regressions/M18882/test-config.yaml 3259 
>   /asterisk/trunk/tests/rfc2833_dtmf_detect/test-config.yaml 3259 
>   /asterisk/trunk/tests/skeleton_test/test-config.yaml 3259 
>   /asterisk/trunk/tests/udptl/test-config.yaml 3259 
>   /asterisk/trunk/tests/udptl_v6/test-config.yaml 3259 
> 
> Diff: https://reviewboard.asterisk.org/r/1982/diff
> 
> 
> Testing
> -------
> 
> Ran with:
> 1.8 branch
> 10 branch
> 10-digiumphones branch
> 1.8.11-cert3 tag
> trunk
> some team branch of mine
> 
> The tests that were supposed to run against a version did; those that weren't, didn't.
> 
> 
> Thanks,
> 
> Matt
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20120615/ca13e961/attachment-0001.htm>

More information about the asterisk-dev mailing list