Script to find the latest Branches on GIT Repository

 For every release we create a release branch for every microservice. few times we miss few microservice and it'll fail the deployment. hence i created this script to find the latest release branch on each repository to ensure for a smooth deployment.

1. We need to create reference file with GIT remote URL. we can copy the clone SSH git from each repo and generate this file

>> cat branch.txt

git@github.abc.com:DSC/a-frontend.git

git@github.abc.com:DSC/b-service.git

git@github.abc.com:DSC/c-portal.git

git@github.abc.com:DSC/d-service.git

git@github.abc.com:DSC/e-service.git

git@github.abc.com:DSC/f-service.git

git@github.abc.com:DSC/g-service.git

git@github.abc.com:DSC/h-service.git

git@github.abc.com:DSC/i-service.git

git@github.abc.com:DSC/j-service.git

2. We need to create the script,

>>vi find_branch.sh

#!/bin/bash

# Script to pull the GIT Repositories and find the latest branch"

echo -e "       Summary    " > latest.txt

echo -e "-------------------" >> latest.txt

for i in `cat branch.txt`;

  do

   j=`echo $i | cut -d/ -f2 | cut -d. -f1`

   echo "###################################"

   echo "Sync the $j Git Repository"

   echo "###################################"

   #`git clone $i --recurse-submodules`

   cd $j

   git pull --recurse-submodules

   git branch -r --sort=-committerdate | grep release | head -2

   echo "$j = `git branch -r --sort=-committerdate | grep release | head -1`" >> ../latest.txt

   cd ../

   echo -e "###################################\n\n"

done

cat latest.txt

3. Execute the create the script,

>>./find_branch.sh

Grep options

 To grep two words on a file 

            grep -E 'RoleName|RoleId' /tmp/roles.txt

            grep -e RoleName -e RoleId /tmp/roles.txt

To grep Before and after lines from a file

            grep -B1 -A1 RoleId /tmp/roles.txt

To find the IAM Roles on a AWS account

        aws iam list-roles > /tmp/roles.txt

 

Switch User(SU) without Password

 Switch user without password.

    In my case, i've got two users on same machine and whenever i do su - <user> is still asking password  and we'll not achieve automation on this manner.

      Hence the solution is 

#> sudo -u <user> -i

    This command will take us directly into the user without prompting us with password.

SHELL: sed to search and replace the Variable

 Usually we search and replace a string on a file using SED editor. whereas in my scenario, i need to replace the string with the Variable on the Shell script. 

OLD_VAR = "blah blah"

NEW_VAR = "xyz"

   sed -i 's|'"$OLD_VAR"'|'$NEW_VAR'|g' temp-file.txt

Refer : http://joe-re.blogspot.com/2020/08/aws-extract-zip-from-one-bucket-to.html

AWS: Extract a ZIP from one bucket to another bucket

 Below is the Script which help me to Extract the ZIP from S3 Bucket from one AWS Account to another S3 account.

This particular Script will do the following things,

• List the Files on S3 Bucket and find the latest Build/file,
• Download the ZIP file form S3 Bucket of different account,
• Extract the ZIP and update the server-config.json,
• Replace the string with SED command,
• Upload the Extracted folder to different S3 bucket. 

# Download the AdminPortal zip

pwd

export ZIPNAME=`aws s3 ls ${S3_Source_Bucket} --profile=apportal-dev | grep AdminPortal | cut -d" " -f7 | tail -1`

echo ${ZIPNAME}

export ZIPVER=`echo ${ZIPNAME} | cut -d"." -f2-5`

Download_S3 ()

{

  echo "================================================================="

  echo "Downloading the UI Zip from APP-Deliveries Bucket"

  echo "================================================================="

  #aws s3 cp s3://s3-app-dev-deliveries/AdminPortal.1.0.0.1.zip .

  aws s3 cp s3://${S3_Source_Bucket}/${ZIPNAME} .

  echo "================================================================="

  echo "uncompressing the Zip"

  echo "================================================================="

  unzip ${ZIPNAME}

  

  echo "================================================================="

  echo "Updating the URLs"

  echo "================================================================="

  cd ap-ap

  cp server-config.json server-config.json.bkp

  cp server-config.json temp-file.txt

   OLD_AAS=AASSERVER

   NEW_AAS=${AASSERVER}

   OLD_LAMB=LAMBDASERVER

   NEW_LAMB=${LAMBDASERVER}

   OLD_SSP=SSPSERVER

   NEW_SSP=${SSPSERVER}

   

   sed -i 's|'"$OLD_AAS"'|'$NEW_AAS'|g' temp-file.txt

   sed -i 's|'"$OLD_LAMB"'|'$NEW_LAMB'|g' temp-file.txt

   sed -i 's|'"$OLD_SSP"'|'$NEW_SSP'|g' temp-file.txt

   #perl -i -p -e 's/AASSERVER/$AASSERVER/g' temp-file.txt

   #perl -i -p -e 's/LAMBDASERVER/$LAMBDASERVER/g' temp-file.txt

 

  mv temp-file.txt server-config.json

  cat server-config.json

  echo "https://s3-app-dev-deliveries.s3.amazonaws.com/${ZIPNAME}" > version-${ZIPVER}.txt

}

Upload_S3 ()

{

  echo "================================================================="

  echo "Uploading the Static Website to the UI Bucket"

  echo "================================================================="

  cd ..

  #aws s3 cp ap-ap s3://${S3_Dest_Bucket} --recursive

  aws s3 sync ap-ap s3://${S3_Dest_Bucket} --delete --exclude '*.svg'

  aws s3 sync ap-ap s3://${S3_Dest_Bucket} --delete --exclude '*' --include '*.svg' --content-type 'image/svg+xml'

}

Empty_S3 ()

{

 aws s3 rm s3://${S3_Dest_Bucket} --recursive

 echo "${S3_Dest_Bucket} Bucket is emptied"

}

Update_ver ()

{

 cd ${WORKSPACE}/ap-ap

 cp /var/opt/hid/app-version/background.jpg /var/opt/hid/app-version/version.html .

  sed -i 's|'PLAT'|'${ENV}'|g' version.html

  sed -i 's|'UPDATEVER'|'${ZIPVER}'|g' version.html

}

if [ "$EMPTY_BUCKET" == "yes" ]; then

    echo "Empty the S3 and Uploading the new Static Website."

    Empty_S3

    Download_S3

    Update_ver

    Upload_S3

else

   echo "Uploading to S3 Without Clearing S3"

   Download_S3

   Update_ver

   Upload_S3

fi

SHELL: Add a line to a file

 If we want to add a line to a file as specific place, please use this command,

sed -i '1i Inserting at First Line' sample.txt

sed -i '4i "Inserting" : "At Fourth Line" ' sample.txt
With these commands we can insert the mentioned parameter at the mentioned line on the original file

Convert Unicode file in Linux

Scenario:
          I got a file, when we open it using VI editor. i found <96> & <93> characters instead of - & "". none of the utils helped me open the file as it is.

Solution:

• Get revision / open the file in any editor,
• Copy the contents and paste it in WORD (Best decryptor of any format of content)
• Save the file as .doc,
• Open the .doc and copy the contents,
• Open the VI editor in the linux box and paste the contents and save the file
• We got the file as the orignal notations.

Issue:

      Developers used to convert the pdf to txt using some utility, due to this we can't open the file with the expected CHARSET. i've tried dos2unix, unix2dos, iconv nothing helped. :(

Install PERL Module Digest::MD5

This Post tells us about to install Digest::MD5 module of perl,

Pre-Requirements,
 We need GCC to install the perl modules into linux, please refer GCC Install Guide link to install GCC.

Steps:

• Download Digest-MD5-2.52.tar.gz from the CPAN,
• Uncompress the file into a Directory,
• Change directory to Digest-MD5-2.52
• [root@castle-hat Digest-MD5-2.52]# cd Digest-MD5-2.52
• Run the Makefile with perl compiler,
• [root@castle-hat Digest-MD5-2.52]# perl Makefile.PL
• Run make -- may ignore some warnings,
• [root@castle-hat Digest-MD5-2.52]# make
• Run Make test, look for All tests successful.& Result : PASS
• [root@castle-hat Digest-MD5-2.52]# make test
• Run Make Install
• [root@castle-hat Digest-MD5-2.52]# make install
• To check whether the module properly installed / not, (if prompt comes without exception, then it means installed on the machine)
• [root@castle-hat Digest-MD5-2.52]#  perl -e 'use Digest::MD5;'
• [root@castle-hat Digest-MD5-2.52]#

Grep Wildcarded data from a file

Grepping a Word with numbers in a file

Scenario:
cat SHAMD5.txt
df58a5e677a3e0f4a417744ad3cbb7994fe57ae7        60827   ./lib/ims-systemfields-1.0.1.1.1330974.jar

Failure results:

grep '/lib/ims-systemfields-*.jar' SHAMD5.txt

   -- Doesn't give the output because there is only one occurrence of digit is replaced with  *

Success results:

grep '/lib/ims-systemfields-.*jar' SHAMD5.txt

   -- Gives the expected output because .* represents anything upto jar

Better Successful results:

grep '/lib/ims-systemfields-*.*jar' SHAMD5.txt

   -- Gives the exact expected output because *.* never mind any occurances