<p>George Joseph has uploaded this change for <strong>review</strong>.</p><p><a href="https://gerrit.asterisk.org/c/asterisk/+/19945">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">res_pjsip: Replace invalid UTF-8 sequences in callerid name<br><br>* Added a new function ast_utf8_replace_invalid_chars() to<br> utf8.c that copies a string replacing any invalid UTF-8<br> sequences with the Unicode specified U+FFFD replacement<br> character. For example: "abc\xffdef" becomes "abc\uFFFDdef".<br> Any UTF-8 compliant implementation will show that character<br> as a � character.<br><br>* Updated res_pjsip:set_id_from_hdr() to use<br> ast_utf8_replace_invalid_chars and print a warning if any<br> invalid sequences were found during the copy.<br><br>* Updated stasis_channels:ast_channel_publish_varset to use<br> ast_utf8_replace_invalid_chars and print a warning if any<br> invalid sequences were found during the copy.<br><br>ASTERISK-27830<br><br>Change-Id: I4ffbdb19c80bf0efc675d40078a3ca4f85c567d8<br>---<br>M include/asterisk/utf8.h<br>M main/stasis_channels.c<br>M main/utf8.c<br>M res/res_pjsip.c<br>4 files changed, 673 insertions(+), 3 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://gerrit.asterisk.org:29418/asterisk refs/changes/45/19945/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/include/asterisk/utf8.h b/include/asterisk/utf8.h</span><br><span>index 02ec800..6e6a89d 100644</span><br><span>--- a/include/asterisk/utf8.h</span><br><span>+++ b/include/asterisk/utf8.h</span><br><span>@@ -67,6 +67,59 @@</span><br><span> */</span><br><span> void ast_utf8_copy_string(char *dst, const char *src, size_t size);</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+enum ast_utf8_replace_result {</span><br><span style="color: hsl(120, 100%, 40%);">+ /*! \brief Source contained fully valid UTF-8</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * The entire string was valid UTF-8 and no replacement</span><br><span style="color: hsl(120, 100%, 40%);">+ * was required.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ AST_UTF8_REPLACE_VALID,</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*! \brief Source contained at least 1 invalid UTF-8 sequence</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * Parts of the string contained invalid UTF-8 sequences</span><br><span style="color: hsl(120, 100%, 40%);">+ * but those were successfully replaced with the U+FFFD</span><br><span style="color: hsl(120, 100%, 40%);">+ * replacement sequence.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ AST_UTF8_REPLACE_INVALID,</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*! \brief Not enough space to copy entire source</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * The destination buffer wasn't large enough to copy</span><br><span style="color: hsl(120, 100%, 40%);">+ * all of the source characters. As many of the source</span><br><span style="color: hsl(120, 100%, 40%);">+ * characters that could be copied/replaced were done so</span><br><span style="color: hsl(120, 100%, 40%);">+ * and a final NULL terminator added.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ AST_UTF8_REPLACE_OVERRUN,</span><br><span style="color: hsl(120, 100%, 40%);">+};</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \brief Copy a string safely replacing any invalid UTF-8 sequences</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * This is similar to \ref ast_copy_string, but it will only copy valid UTF-8</span><br><span style="color: hsl(120, 100%, 40%);">+ * sequences from the source string into the destination buffer.</span><br><span style="color: hsl(120, 100%, 40%);">+ * If an invalid sequence is encountered, it's replaced with the \uFFFD</span><br><span style="color: hsl(120, 100%, 40%);">+ * sequence which is the valid UTF-8 sequence that represents an unknown,</span><br><span style="color: hsl(120, 100%, 40%);">+ * unrecognized, or unrepresentable character. Since \uFFFD is actually a</span><br><span style="color: hsl(120, 100%, 40%);">+ * 3 byte sequence, the destination buffer will need to be larger than</span><br><span style="color: hsl(120, 100%, 40%);">+ * the corresponding source string if it contains invalid sequences.</span><br><span style="color: hsl(120, 100%, 40%);">+ * You can pass NULL as the destination buffer pointer to get the actual</span><br><span style="color: hsl(120, 100%, 40%);">+ * size required, then call the function again with the properly sized</span><br><span style="color: hsl(120, 100%, 40%);">+ * buffer.</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * \param dst Pointer to the destination buffer. If NULL,</span><br><span style="color: hsl(120, 100%, 40%);">+ * dst_size will be set to the size of the</span><br><span style="color: hsl(120, 100%, 40%);">+ * buffer required to fully process the</span><br><span style="color: hsl(120, 100%, 40%);">+ * source string.</span><br><span style="color: hsl(120, 100%, 40%);">+ * \param dst_size A pointer to the size of the dst buffer</span><br><span style="color: hsl(120, 100%, 40%);">+ * \param src The source string</span><br><span style="color: hsl(120, 100%, 40%);">+ * \param src_len The number of bytes to copy</span><br><span style="color: hsl(120, 100%, 40%);">+ *</span><br><span style="color: hsl(120, 100%, 40%);">+ * \return \ref ast_utf8_replace_result</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+enum ast_utf8_replace_result ast_utf8_replace_invalid_chars(char *dst,</span><br><span style="color: hsl(120, 100%, 40%);">+ size_t *dst_size, const char *src, size_t src_len);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> enum ast_utf8_validation_result {</span><br><span> /*! \brief The consumed sequence is valid UTF-8</span><br><span> *</span><br><span>diff --git a/main/stasis_channels.c b/main/stasis_channels.c</span><br><span>index d373f6a..255ee2a 100644</span><br><span>--- a/main/stasis_channels.c</span><br><span>+++ b/main/stasis_channels.c</span><br><span>@@ -39,6 +39,7 @@</span><br><span> #include "asterisk/stasis_channels.h"</span><br><span> #include "asterisk/dial.h"</span><br><span> #include "asterisk/linkedlists.h"</span><br><span style="color: hsl(120, 100%, 40%);">+#include "asterisk/utf8.h"</span><br><span> </span><br><span> /*** DOCUMENTATION</span><br><span> <managerEvent language="en_US" name="VarSet"></span><br><span>@@ -1154,13 +1155,43 @@</span><br><span> void ast_channel_publish_varset(struct ast_channel *chan, const char *name, const char *value)</span><br><span> {</span><br><span> struct ast_json *blob;</span><br><span style="color: hsl(120, 100%, 40%);">+ enum ast_utf8_replace_result result;</span><br><span style="color: hsl(120, 100%, 40%);">+ char *new_value = NULL;</span><br><span style="color: hsl(120, 100%, 40%);">+ size_t new_value_size = 0;</span><br><span> </span><br><span> ast_assert(name != NULL);</span><br><span> ast_assert(value != NULL);</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Call with new-value == NULL to just check for invalid UTF-8</span><br><span style="color: hsl(120, 100%, 40%);">+ * sequences and get size of buffer needed.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ result = ast_utf8_replace_invalid_chars(new_value, &new_value_size,</span><br><span style="color: hsl(120, 100%, 40%);">+ value, strlen(value));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (result == AST_UTF8_REPLACE_VALID) {</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * If there were no invalid sequences, we can use</span><br><span style="color: hsl(120, 100%, 40%);">+ * the value directly.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ new_value = (char *)value;</span><br><span style="color: hsl(120, 100%, 40%);">+ } else {</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * If there were invalid sequences, we need to replace</span><br><span style="color: hsl(120, 100%, 40%);">+ * them with the UTF-8 U+FFFD replacement character.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ new_value = ast_alloca(new_value_size);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ result = ast_utf8_replace_invalid_chars(new_value, &new_value_size,</span><br><span style="color: hsl(120, 100%, 40%);">+ value, strlen(value));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_log(LOG_WARNING, "%s: The contents of variable '%s' had invalid UTF-8 sequences which were replaced",</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_channel_name(chan), name);</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> blob = ast_json_pack("{s: s, s: s}",</span><br><span> "variable", name,</span><br><span style="color: hsl(0, 100%, 40%);">- "value", value);</span><br><span style="color: hsl(120, 100%, 40%);">+ "value", new_value);</span><br><span> if (!blob) {</span><br><span> ast_log(LOG_ERROR, "Error creating message\n");</span><br><span> return;</span><br><span>diff --git a/main/utf8.c b/main/utf8.c</span><br><span>index ab4338b..00015ed 100644</span><br><span>--- a/main/utf8.c</span><br><span>+++ b/main/utf8.c</span><br><span>@@ -156,6 +156,154 @@</span><br><span> *last_good = '\0';</span><br><span> }</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+/*!</span><br><span style="color: hsl(120, 100%, 40%);">+ * \warning A UTF-8 sequence could be 1, 2, 3 or 4 bytes long depending</span><br><span style="color: hsl(120, 100%, 40%);">+ * on the first byte in the sequence. Don't try to modify this function</span><br><span style="color: hsl(120, 100%, 40%);">+ * without understanding how UTF-8 works.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*</span><br><span style="color: hsl(120, 100%, 40%);">+ * The official unicode replacement character is U+FFFD</span><br><span style="color: hsl(120, 100%, 40%);">+ * which is actually the 3 following bytes:</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+#define REPL_SEQ "\xEF\xBF\xBD"</span><br><span style="color: hsl(120, 100%, 40%);">+#define REPL_SEQ_LEN 3</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+enum ast_utf8_replace_result</span><br><span style="color: hsl(120, 100%, 40%);">+ast_utf8_replace_invalid_chars(char *dst, size_t *dst_size, const char *src,</span><br><span style="color: hsl(120, 100%, 40%);">+ size_t src_len)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ enum ast_utf8_replace_result res = AST_UTF8_REPLACE_VALID;</span><br><span style="color: hsl(120, 100%, 40%);">+ size_t src_pos = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+ size_t dst_pos = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+ uint32_t prev_state = UTF8_ACCEPT;</span><br><span style="color: hsl(120, 100%, 40%);">+ uint32_t curr_state = UTF8_ACCEPT;</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * UTF-8 sequences can be 1 - 4 bytes in length so we</span><br><span style="color: hsl(120, 100%, 40%);">+ * have to keep track of where we are.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ int seq_len = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (dst) {</span><br><span style="color: hsl(120, 100%, 40%);">+ memset(dst, 0, *dst_size);</span><br><span style="color: hsl(120, 100%, 40%);">+ } else {</span><br><span style="color: hsl(120, 100%, 40%);">+ *dst_size = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (!src || src_len == 0) {</span><br><span style="color: hsl(120, 100%, 40%);">+ return AST_UTF8_REPLACE_VALID;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for (prev_state = 0, curr_state = 0; src_pos < src_len; prev_state = curr_state, src_pos++) {</span><br><span style="color: hsl(120, 100%, 40%);">+ uint32_t rc;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ rc = decode(&curr_state, (uint8_t) src[src_pos]);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (dst && dst_pos >= *dst_size - 1) {</span><br><span style="color: hsl(120, 100%, 40%);">+ if (prev_state > UTF8_REJECT) {</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * We ran out of space in the middle of a possible</span><br><span style="color: hsl(120, 100%, 40%);">+ * multi-byte sequence so we have to back up and</span><br><span style="color: hsl(120, 100%, 40%);">+ * overwrite the start of the sequence with the</span><br><span style="color: hsl(120, 100%, 40%);">+ * NULL terminator.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ dst_pos -= (seq_len - (prev_state / 36));</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ dst[dst_pos] = '\0';</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return AST_UTF8_REPLACE_OVERRUN;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (rc == UTF8_ACCEPT) {</span><br><span style="color: hsl(120, 100%, 40%);">+ if (dst) {</span><br><span style="color: hsl(120, 100%, 40%);">+ dst[dst_pos] = src[src_pos];</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ dst_pos++;</span><br><span style="color: hsl(120, 100%, 40%);">+ seq_len = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (rc > UTF8_REJECT) {</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * We're possibly at the start of, or in the middle of,</span><br><span style="color: hsl(120, 100%, 40%);">+ * a multi-byte sequence. The curr_state will tell us how many</span><br><span style="color: hsl(120, 100%, 40%);">+ * bytes _should_ be remaining in the sequence.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ if (prev_state == UTF8_ACCEPT) {</span><br><span style="color: hsl(120, 100%, 40%);">+ /* If the previous state was a good character then</span><br><span style="color: hsl(120, 100%, 40%);">+ * this can only be the start of s sequence</span><br><span style="color: hsl(120, 100%, 40%);">+ * which is all we care about.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ seq_len = curr_state / 36 + 1;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (dst) {</span><br><span style="color: hsl(120, 100%, 40%);">+ dst[dst_pos] = src[src_pos];</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ dst_pos++;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (rc == UTF8_REJECT) {</span><br><span style="color: hsl(120, 100%, 40%);">+ /* We got at least 1 rejection so the string is invalid */</span><br><span style="color: hsl(120, 100%, 40%);">+ res = AST_UTF8_REPLACE_INVALID;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (prev_state != UTF8_ACCEPT) {</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * If we were in a multi-byte sequence and this</span><br><span style="color: hsl(120, 100%, 40%);">+ * byte isn't valid at this time, we'll back</span><br><span style="color: hsl(120, 100%, 40%);">+ * the destination pointer back to the start</span><br><span style="color: hsl(120, 100%, 40%);">+ * of the now-invalid sequence and write the</span><br><span style="color: hsl(120, 100%, 40%);">+ * replacement bytes there. Then we'll</span><br><span style="color: hsl(120, 100%, 40%);">+ * process the current byte again in the next</span><br><span style="color: hsl(120, 100%, 40%);">+ * loop iteration. It may be quite valid later.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ dst_pos -= (seq_len - (prev_state / 36));</span><br><span style="color: hsl(120, 100%, 40%);">+ src_pos--;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ if (dst) {</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * If we're not just calculating the needed destination</span><br><span style="color: hsl(120, 100%, 40%);">+ * buffer space, and we don't have enough room to write</span><br><span style="color: hsl(120, 100%, 40%);">+ * the replacement sequence, terminate the output</span><br><span style="color: hsl(120, 100%, 40%);">+ * and return.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ if (dst_pos > *dst_size - 4) {</span><br><span style="color: hsl(120, 100%, 40%);">+ dst[dst_pos] = '\0';</span><br><span style="color: hsl(120, 100%, 40%);">+ return AST_UTF8_REPLACE_OVERRUN;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ memcpy(&dst[dst_pos], REPL_SEQ, REPL_SEQ_LEN);</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ dst_pos += REPL_SEQ_LEN;</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Reset the state machine */</span><br><span style="color: hsl(120, 100%, 40%);">+ curr_state = UTF8_ACCEPT;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (curr_state != UTF8_ACCEPT) {</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * We were probably in the middle of a</span><br><span style="color: hsl(120, 100%, 40%);">+ * sequence and ran out of space.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ res = AST_UTF8_INVALID;</span><br><span style="color: hsl(120, 100%, 40%);">+ dst_pos -= (seq_len - (prev_state / 36));</span><br><span style="color: hsl(120, 100%, 40%);">+ if (dst) {</span><br><span style="color: hsl(120, 100%, 40%);">+ if (dst_pos > *dst_size - 4) {</span><br><span style="color: hsl(120, 100%, 40%);">+ dst[dst_pos] = '\0';</span><br><span style="color: hsl(120, 100%, 40%);">+ return AST_UTF8_REPLACE_OVERRUN;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ memcpy(&dst[dst_pos], REPL_SEQ, REPL_SEQ_LEN);</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+ dst_pos += REPL_SEQ_LEN;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (dst) {</span><br><span style="color: hsl(120, 100%, 40%);">+ dst[dst_pos] = '\0';</span><br><span style="color: hsl(120, 100%, 40%);">+ } else {</span><br><span style="color: hsl(120, 100%, 40%);">+ *dst_size = dst_pos + 1;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return res;</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> struct ast_utf8_validator {</span><br><span> uint32_t state;</span><br><span> };</span><br><span>@@ -219,6 +367,8 @@</span><br><span> </span><br><span> #ifdef TEST_FRAMEWORK</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+#include "asterisk/json.h"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> AST_TEST_DEFINE(test_utf8_is_valid)</span><br><span> {</span><br><span> switch (cmd) {</span><br><span>@@ -313,6 +463,398 @@</span><br><span> return AST_TEST_PASS;</span><br><span> }</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+/*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Let the replace function determine how much</span><br><span style="color: hsl(120, 100%, 40%);">+ * buffer space is required for the destination.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+#define SIZE_REQUIRED 0</span><br><span style="color: hsl(120, 100%, 40%);">+/*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Set the destination buffer size to the size</span><br><span style="color: hsl(120, 100%, 40%);">+ * we expect it to be. 0xDead has no meaning</span><br><span style="color: hsl(120, 100%, 40%);">+ * other than it's larger than any test needs</span><br><span style="color: hsl(120, 100%, 40%);">+ * a buffer to be.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+#define SIZE_EXPECTED 0xDead</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+static int tracs(int run, const char *src, const char *cmp,</span><br><span style="color: hsl(120, 100%, 40%);">+ size_t dst_size, enum ast_utf8_replace_result exp_result)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ char *dst = NULL;</span><br><span style="color: hsl(120, 100%, 40%);">+ struct ast_json *blob;</span><br><span style="color: hsl(120, 100%, 40%);">+ enum ast_utf8_replace_result result;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (dst_size == SIZE_REQUIRED) {</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_utf8_replace_invalid_chars(dst, &dst_size, src, src ? strlen(src) : 0);</span><br><span style="color: hsl(120, 100%, 40%);">+ } else if (dst_size == SIZE_EXPECTED) {</span><br><span style="color: hsl(120, 100%, 40%);">+ dst_size = strlen(cmp) + 1;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ dst = (char *)ast_alloca(dst_size);</span><br><span style="color: hsl(120, 100%, 40%);">+ result = ast_utf8_replace_invalid_chars(dst, &dst_size, src, src ? strlen(src) : 0);</span><br><span style="color: hsl(120, 100%, 40%);">+ if (result != exp_result || strcmp(dst, cmp) != 0) {</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_log(LOG_ERROR, "Run: %2d Invalid result. Src: '%s', Dst: '%s', ExpDst: '%s' Result: %d ExpResult: %d\n",</span><br><span style="color: hsl(120, 100%, 40%);">+ run, src, dst, cmp, result, exp_result);</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * The ultimate test: Does jansson accept the result as valid UTF-8?</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ blob = ast_json_pack("{s: s, s: s}",</span><br><span style="color: hsl(120, 100%, 40%);">+ "variable", "doesntmatter",</span><br><span style="color: hsl(120, 100%, 40%);">+ "value", dst);</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_json_unref(blob);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return blob != NULL;</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+#define ATV(t, v) ast_test_validate(t, v)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+AST_TEST_DEFINE(test_utf8_replace_invalid_chars)</span><br><span style="color: hsl(120, 100%, 40%);">+{</span><br><span style="color: hsl(120, 100%, 40%);">+ const char *src;</span><br><span style="color: hsl(120, 100%, 40%);">+ size_t dst_size;</span><br><span style="color: hsl(120, 100%, 40%);">+ enum ast_utf8_replace_result result;</span><br><span style="color: hsl(120, 100%, 40%);">+ int k = 0;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ switch (cmd) {</span><br><span style="color: hsl(120, 100%, 40%);">+ case TEST_INIT:</span><br><span style="color: hsl(120, 100%, 40%);">+ info->name = "replace_invalid";</span><br><span style="color: hsl(120, 100%, 40%);">+ info->category = "/main/utf8/";</span><br><span style="color: hsl(120, 100%, 40%);">+ info->summary = "Test ast_utf8_replace_invalid_chars";</span><br><span style="color: hsl(120, 100%, 40%);">+ info->description =</span><br><span style="color: hsl(120, 100%, 40%);">+ "Tests UTF-8 string copying/replacing code.";</span><br><span style="color: hsl(120, 100%, 40%);">+ return AST_TEST_NOT_RUN;</span><br><span style="color: hsl(120, 100%, 40%);">+ case TEST_EXECUTE:</span><br><span style="color: hsl(120, 100%, 40%);">+ break;</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+/*</span><br><span style="color: hsl(120, 100%, 40%);">+ Table 3-7. Well-Formed UTF-8 Byte Sequences</span><br><span style="color: hsl(120, 100%, 40%);">+ Code Points First Second Third Fourth</span><br><span style="color: hsl(120, 100%, 40%);">+ Byte Byte Byte Byte</span><br><span style="color: hsl(120, 100%, 40%);">+ U+0000..U+007F 00..7F</span><br><span style="color: hsl(120, 100%, 40%);">+ U+0080..U+07FF C2..DF 80..BF</span><br><span style="color: hsl(120, 100%, 40%);">+ U+0800..U+0FFF E0 A0..BF 80..BF</span><br><span style="color: hsl(120, 100%, 40%);">+ U+1000..U+CFFF E1..EC 80..BF 80..BF</span><br><span style="color: hsl(120, 100%, 40%);">+ U+D000..U+D7FF ED 80..9F 80..BF</span><br><span style="color: hsl(120, 100%, 40%);">+ U+E000..U+FFFF EE..EF 80..BF 80..BF</span><br><span style="color: hsl(120, 100%, 40%);">+ U+10000..U+3FFFF F0 90..BF 80..BF 80..BF</span><br><span style="color: hsl(120, 100%, 40%);">+ U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF</span><br><span style="color: hsl(120, 100%, 40%);">+ U+100000..U+10FFFF F4 80..8F 80..BF 80..BF</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ Older compilers don't support using the \uXXXX or \UXXXXXXXX</span><br><span style="color: hsl(120, 100%, 40%);">+ universal character notation so we have to manually specify</span><br><span style="color: hsl(120, 100%, 40%);">+ the byte sequences even for valid UTF-8 sequences.</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ These are the ones used for the tests below:</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ \u00B0 = \xC2\xB0</span><br><span style="color: hsl(120, 100%, 40%);">+ \u0800 = \xE0\xA0\x80</span><br><span style="color: hsl(120, 100%, 40%);">+ \uE000 = \xEE\x80\x80</span><br><span style="color: hsl(120, 100%, 40%);">+ \U00040000 = \xF1\x80\x80\x80</span><br><span style="color: hsl(120, 100%, 40%);">+*/</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Check that NULL destination with a valid source string gives us a</span><br><span style="color: hsl(120, 100%, 40%);">+ * valid result code and buffer size = the length of the input string</span><br><span style="color: hsl(120, 100%, 40%);">+ * plus room for the NULL terminator.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ src = "ABC\xC2\xB0xyz";</span><br><span style="color: hsl(120, 100%, 40%);">+ result = ast_utf8_replace_invalid_chars(NULL, &dst_size, src, src ? strlen(src) : 0);</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, result == AST_UTF8_REPLACE_VALID && dst_size == strlen(src) + 1);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Check that NULL destination with an invalid source string gives us an</span><br><span style="color: hsl(120, 100%, 40%);">+ * invalid result code and buffer size = the length of the input string</span><br><span style="color: hsl(120, 100%, 40%);">+ * plus room for the NULL terminator plus the 2 extra bytes needed for</span><br><span style="color: hsl(120, 100%, 40%);">+ * the one replacement character.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ src = "ABC\xFFxyz";</span><br><span style="color: hsl(120, 100%, 40%);">+ result = ast_utf8_replace_invalid_chars(NULL, &dst_size, src, src ? strlen(src) : 0);</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, result == AST_UTF8_REPLACE_INVALID && dst_size == strlen(src) + 3);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * NULL or empty input</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, NULL, "", 80, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "", "", 80, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Let the replace function calculate the space needed for result */</span><br><span style="color: hsl(120, 100%, 40%);">+ k = SIZE_REQUIRED;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Basic ASCII string</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyzA", "ABC xyzA", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Mid string.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good single sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0xyz", "ABC\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80xyz", "ABC\xE0\xA0\x80xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xF1\x80\x80\x80xyz", "ABC\xF1\x80\x80\x80xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good multiple adjacent sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0\xC2\xB0xyz", "ABC\xC2\xB0\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80\xC2\xB0xyz", "ABC\xE0\xA0\x80\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xF1\x80\x80\x80\xC2\xB0xyz", "ABC\xF1\x80\x80\x80\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Bad sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2xyz", "ABC\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xC2xyz", "ABC\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xB0xyz", "ABC\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\xC2xyz", "ABC\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\xF5xyz", "ABC\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0xyz", "ABC\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Beginning of string.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good single sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xC2\xB0xyz", "\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xE0\xA0\x80xyz", "\xE0\xA0\x80xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xF1\x80\x80\x80xyz", "\xF1\x80\x80\x80xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good multiple adjacent sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xC2\xB0\xC2\xB0xyz", "\xC2\xB0\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xE0\xA0\x80\xC2\xB0xyz", "\xE0\xA0\x80\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xF1\x80\x80\x80\xC2\xB0xyz", "\xF1\x80\x80\x80\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Bad sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xC2xyz", "\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xC2\xC2xyz", "\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xB0xyz", "\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xE0\xA0\xC2xyz", "\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xE0\xA0\xF5xyz", "\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xE0\xA0xyz", "\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * End of string.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good single sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0", "ABC\xC2\xB0", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80", "ABC\xE0\xA0\x80", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xF1\x80\x80\x80", "ABC\xF1\x80\x80\x80", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good multiple adjacent sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0\xC2\xB0", "ABC\xC2\xB0\xC2\xB0", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80\xC2\xB0", "ABC\xE0\xA0\x80\xC2\xB0", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xF1\x80\x80\x80\xC2\xB0", "ABC\xF1\x80\x80\x80\xC2\xB0", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Bad sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2", "ABC\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xC2", "ABC\xEF\xBF\xBD\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xB0", "ABC\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\xC2", "ABC\xEF\xBF\xBD\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\xF5", "ABC\xEF\xBF\xBD\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0", "ABC\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Force destination buffer to be only large enough to hold the expected result */</span><br><span style="color: hsl(120, 100%, 40%);">+ k = SIZE_EXPECTED;</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Mid string.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good single sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0xyz", "ABC\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80xyz", "ABC\xE0\xA0\x80xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xF1\x80\x80\x80xyz", "ABC\xF1\x80\x80\x80xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good multiple adjacent sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0\xC2\xB0xyz", "ABC\xC2\xB0\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80\xC2\xB0xyz", "ABC\xE0\xA0\x80\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xF1\x80\x80\x80\xC2\xB0xyz", "ABC\xF1\x80\x80\x80\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Bad sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2xyz", "ABC\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xC2xyz", "ABC\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xB0xyz", "ABC\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\xC2xyz", "ABC\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\xF5xyz", "ABC\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0xyz", "ABC\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Beginning of string.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good single sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xC2\xB0xyz", "\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xE0\xA0\x80xyz", "\xE0\xA0\x80xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xF1\x80\x80\x80xyz", "\xF1\x80\x80\x80xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good multiple adjacent sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xC2\xB0\xC2\xB0xyz", "\xC2\xB0\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xE0\xA0\x80\xC2\xB0xyz", "\xE0\xA0\x80\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xF1\x80\x80\x80\xC2\xB0xyz", "\xF1\x80\x80\x80\xC2\xB0xyz", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Bad sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xC2xyz", "\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xC2\xC2xyz", "\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xB0xyz", "\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xE0\xA0\xC2xyz", "\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xE0\xA0\xF5xyz", "\xEF\xBF\xBD\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "\xE0\xA0xyz", "\xEF\xBF\xBDxyz", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * End of string.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good single sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0", "ABC\xC2\xB0", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80", "ABC\xE0\xA0\x80", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xF1\x80\x80\x80", "ABC\xF1\x80\x80\x80", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good multiple adjacent sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0\xC2\xB0", "ABC\xC2\xB0\xC2\xB0", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80\xC2\xB0", "ABC\xE0\xA0\x80\xC2\xB0", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xF1\x80\x80\x80\xC2\xB0", "ABC\xF1\x80\x80\x80\xC2\xB0", k, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ /* Bad sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2", "ABC\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xC2", "ABC\xEF\xBF\xBD\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xB0", "ABC\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\xC2", "ABC\xEF\xBF\xBD\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\xF5", "ABC\xEF\xBF\xBD\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0", "ABC\xEF\xBF\xBD", k, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * Overrun Prevention</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* No frills. */</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 9;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyzA", "ABC xyzA", k--, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyzA", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyzA", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /* good single sequences */</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 9; /* \xC2\xB0 needs 2 bytes */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0xyz", "ABC\xC2\xB0xyz", k--, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0xyz", "ABC\xC2\xB0xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0xyz", "ABC\xC2\xB0x", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0xyz", "ABC\xC2\xB0", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0xyz", "ABC", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0xyz", "ABC", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2\xB0xyz", "AB", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 10; /* \xE0\xA0\x80 needs 3 bytes */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80xyz", "ABC\xE0\xA0\x80xyz", k--, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80xyz", "ABC\xE0\xA0\x80xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80xyz", "ABC\xE0\xA0\x80x", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80xyz", "ABC\xE0\xA0\x80", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80xyz", "ABC", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80xyz", "ABC", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80xyz", "ABC", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xE0\xA0\x80xyz", "AB", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 10; /* \xEF\xBF\xBD needs 3 bytes */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2xyz", "ABC\xEF\xBF\xBDxyz", k--, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2xyz", "ABC\xEF\xBF\xBDxy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2xyz", "ABC\xEF\xBF\xBDx", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2xyz", "ABC\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2xyz", "ABC", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2xyz", "ABC", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2xyz", "ABC", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC\xC2xyz", "AB", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 14; /* Each \xEF\xBF\xBD needs 3 bytes */</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xFF", "ABC xyz\xEF\xBF\xBD\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xFF", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xFF", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xFF", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xFF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xFF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xFF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xFF", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xFF", "ABC x", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * The following tests are classed as "Everything including the kitchen sink".</span><br><span style="color: hsl(120, 100%, 40%);">+ * Some tests may be redundant.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 11;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xFF", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xFF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xFF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xFF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xFF", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 11;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xB0", "ABC xyz\xC2\xB0", k--, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xB0", "ABC xyz\xC2\xB0", k--, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xB0", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xB0", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2\xB0", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 11;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xC2", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 12;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xEE\x80\x80", "ABC xyz\xEE\x80\x80", k--, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xEE\x80\x80", "ABC xyz\xEE\x80\x80", k--, AST_UTF8_REPLACE_VALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xEE\x80\x80", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xEE\x80\x80", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xEE\x80\x80", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xEE\x80\x80", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 11;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 14;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xBF", "ABC xyz\xEF\xBF\xBD\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xBF", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xBF", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xBF", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xBF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xBF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xBF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xBF", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xBF", "ABC x", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 14;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xFF", "ABC xyz\xEF\xBF\xBD\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xFF", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xFF", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xFF", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xFF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xFF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xFF", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xFF", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xFF", "ABC x", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 14;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2", "ABC xyz\xEF\xBF\xBD\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2", "ABC x", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 14;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\x80\xC0", "ABC xyz\xEF\xBF\xBD\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\x80\xC0", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\x80\xC0", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\x80\xC0", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\x80\xC0", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\x80\xC0", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\x80\xC0", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\x80\xC0", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\x80\xC0", "ABC x", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ k = 13;</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2\xB0", "ABC xyz\xEF\xBF\xBD\xC2\xB0", k--, AST_UTF8_REPLACE_INVALID));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2\xB0", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2\xB0", "ABC xyz\xEF\xBF\xBD", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2\xB0", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2\xB0", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2\xB0", "ABC xyz", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+ ATV(test, tracs(__LINE__, "ABC xyz\xED\xC2\xB0", "ABC xy", k--, AST_UTF8_REPLACE_OVERRUN));</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return AST_TEST_PASS;</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> AST_TEST_DEFINE(test_utf8_validator)</span><br><span> {</span><br><span> struct ast_utf8_validator *validator;</span><br><span>@@ -357,6 +899,7 @@</span><br><span> AST_TEST_UNREGISTER(test_utf8_is_valid);</span><br><span> AST_TEST_UNREGISTER(test_utf8_copy_string);</span><br><span> AST_TEST_UNREGISTER(test_utf8_validator);</span><br><span style="color: hsl(120, 100%, 40%);">+ AST_TEST_UNREGISTER(test_utf8_replace_invalid_chars);</span><br><span> }</span><br><span> </span><br><span> int ast_utf8_init(void)</span><br><span>@@ -364,6 +907,7 @@</span><br><span> AST_TEST_REGISTER(test_utf8_is_valid);</span><br><span> AST_TEST_REGISTER(test_utf8_copy_string);</span><br><span> AST_TEST_REGISTER(test_utf8_validator);</span><br><span style="color: hsl(120, 100%, 40%);">+ AST_TEST_REGISTER(test_utf8_replace_invalid_chars);</span><br><span> </span><br><span> ast_register_cleanup(test_utf8_shutdown);</span><br><span> </span><br><span>diff --git a/res/res_pjsip.c b/res/res_pjsip.c</span><br><span>index 8273847..d112aa3 100644</span><br><span>--- a/res/res_pjsip.c</span><br><span>+++ b/res/res_pjsip.c</span><br><span>@@ -47,6 +47,7 @@</span><br><span> #include "asterisk/test.h"</span><br><span> #include "asterisk/res_pjsip_presence_xml.h"</span><br><span> #include "asterisk/res_pjproject.h"</span><br><span style="color: hsl(120, 100%, 40%);">+#include "asterisk/utf8.h"</span><br><span> </span><br><span> /*** MODULEINFO</span><br><span> <depend>pjproject</depend></span><br><span>@@ -2461,12 +2462,12 @@</span><br><span> {</span><br><span> char cid_name[AST_CHANNEL_NAME];</span><br><span> char cid_num[AST_CHANNEL_NAME];</span><br><span style="color: hsl(120, 100%, 40%);">+ size_t cid_name_size = AST_CHANNEL_NAME;</span><br><span> pjsip_name_addr *id_name_addr = (pjsip_name_addr *) hdr->uri;</span><br><span> char *semi;</span><br><span style="color: hsl(120, 100%, 40%);">+ enum ast_utf8_replace_result result;</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- ast_copy_pj_str(cid_name, &id_name_addr->display, sizeof(cid_name));</span><br><span> ast_copy_pj_str(cid_num, ast_sip_pjsip_uri_get_username(hdr->uri), sizeof(cid_num));</span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span> /* Always truncate caller-id number at a semicolon. */</span><br><span> semi = strchr(cid_num, ';');</span><br><span> if (semi) {</span><br><span>@@ -2484,6 +2485,21 @@</span><br><span> *semi = '\0';</span><br><span> }</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ /*</span><br><span style="color: hsl(120, 100%, 40%);">+ * It's safe to pass a NULL or empty string as the source.</span><br><span style="color: hsl(120, 100%, 40%);">+ * The result will be an empty string assuming the destination</span><br><span style="color: hsl(120, 100%, 40%);">+ * size was at least 1.</span><br><span style="color: hsl(120, 100%, 40%);">+ */</span><br><span style="color: hsl(120, 100%, 40%);">+ result = ast_utf8_replace_invalid_chars(cid_name, &cid_name_size,</span><br><span style="color: hsl(120, 100%, 40%);">+ id_name_addr->display.ptr, id_name_addr->display.slen);</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if (result != AST_UTF8_REPLACE_VALID) {</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_log(LOG_WARNING, "CallerID Name '" PJSTR_PRINTF_SPEC</span><br><span style="color: hsl(120, 100%, 40%);">+ "' for number '%s' has invalid UTF-8 characters which "</span><br><span style="color: hsl(120, 100%, 40%);">+ "were replaced",</span><br><span style="color: hsl(120, 100%, 40%);">+ PJSTR_PRINTF_VAR(id_name_addr->display), cid_num);</span><br><span style="color: hsl(120, 100%, 40%);">+ }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> ast_free(id->name.str);</span><br><span> id->name.str = ast_strdup(cid_name);</span><br><span> if (!ast_strlen_zero(cid_name)) {</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.asterisk.org/c/asterisk/+/19945">change 19945</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.asterisk.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.asterisk.org/c/asterisk/+/19945"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: asterisk </div>
<div style="display:none"> Gerrit-Branch: 20 </div>
<div style="display:none"> Gerrit-Change-Id: I4ffbdb19c80bf0efc675d40078a3ca4f85c567d8 </div>
<div style="display:none"> Gerrit-Change-Number: 19945 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: George Joseph <gjoseph@digium.com> </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>