Surprised when sed replacing a token with a URI

Revision history
Tags: linux sed perl

As I was replacing tokens with secrets in an init container for a Kubernetes Pod, I was not getting it to work correctly. Not until I actually took a look at the resulting configuration file, I was getting some hints as to what was wrong. I had used sed before when replacing tokens with URI’s, but apparently not URI’s with multiple query-string parameters which are joined by ampersands (&).

Let’s replace the token %link% with a URI

$ URI=''
$ echo 'my_link = %link%' | sed "s|%link%|${URI}|"
my_link =

Seems good. Let’s add another parameter to the URI

$ URI=''
$ echo 'my_link = %link%' | sed "s|%link%|${URI}|"
my_link =

Surprise. What I expected the link to look like was

A quick search revealed the culprit;

The REPLACEMENT can contain \N (N being a number from 1 to 9, inclusive) references, which refer to the portion of the match which is contained between the Nth \( and its matching \). Also, the REPLACEMENT can contain unescaped & characters which reference the whole matched portion of the pattern space

What I could do here is to first sanitise the URL by backslashing all occurences of & before using it as the replacement

$ echo 'my_link = %link%' | sed "s|%link%|$(echo "$URI" | sed 's|&|\\&|g')|"
my_link =

But to me, that seems clunky. I would rather switch to a different replacer to avoid corner cases like this and excessive code in the init containers. Trying with a perl script instead

$ echo 'my_link = %link%' | perl -p -e "s|%link%|${URI}|"
my_link =

This seems to be okay, but let’s add login credentials and see what happens

$ echo 'my_link = %link%' | perl -p -e "s|%link%|${URI}|"
my_link =

So this time around perl uses @ for named capture groups. In which case I first have to escape them like \@. So I’m where I started out again.

Let’s try with awk

$ echo 'my_link = %link%' | awk "{ gsub(/%link%/, \"$URI\"); print }"
my_link =

Same problem as with sed in that it replaces & with the matching string.

I yield. I found a nice script written by Ed Morton which sanitises both the search and the replacement string before running sed one last time. I added an option to do in-place file replacements.

# Modified version of Ed Morton's sedstr:
# stigok, 2018

# In-place option
if [ "$1" = '-i' ]; then
  opts='-i ' # note trailing space

escOld=$(sed 's/[^^]/[&]/g; s/\^/\\^/g' <<< "$old")
escNew=$(sed 's/[&/\]/\\&/g' <<< "$new")
sed ${opts}"s/$escOld/$escNew/g" "$file"

This is finally a working solution for me with all the possible special characters of the URI in question, without breaking or triggering unwanted features and variable expansions in sed and bash.



If you have any comments or feedback, please send me an e-mail. (stig at stigok dotcom).

Did you find any typos, incorrect information, or have something to add? Then please propose a change to this post.