Recently I've faced a tricky case, related to integration. One e-business system used to update e-payment directory with shell concurrent program and data from Federal Reserve Bank Services web site. System has worked quite well with wget utility and direct link to text file with routing information.
But once it delivered HTML document instead of text document. The reason is quite simple - from now on you should agree with Service terms and then you will get access to the data.
This changes even has led to Github project FedACHDir. Well, with all my respect to GitHub and project authors, I'm not comfortable at all to get such sensitive information from the third party. So I have had to overcome it and automate download.
Let's rethink approach to the task. Well known wget utility allows you to download data form internet but now we have to interact with site, POST information and maintain web session. Answer is simple - curl another well known web utility, included in the most of Linux repositories and compiled for the rest of the operating systems.
Long story short below is a small but elegant Shell script to get data from Federal Reserve Bank Services.
Reminder: use this script means that you are agree with terms and conditions posted on the FRBS site
But once it delivered HTML document instead of text document. The reason is quite simple - from now on you should agree with Service terms and then you will get access to the data.
This changes even has led to Github project FedACHDir. Well, with all my respect to GitHub and project authors, I'm not comfortable at all to get such sensitive information from the third party. So I have had to overcome it and automate download.
Let's rethink approach to the task. Well known wget utility allows you to download data form internet but now we have to interact with site, POST information and maintain web session. Answer is simple - curl another well known web utility, included in the most of Linux repositories and compiled for the rest of the operating systems.
Long story short below is a small but elegant Shell script to get data from Federal Reserve Bank Services.
Reminder: use this script means that you are agree with terms and conditions posted on the FRBS site
#!/bin/sh
#check output file name
if [ -z $1 ]; then
achout=/tmp/FedACHdir.txt
>&2 echo "Save data to /tmp/FedACHdir.txt"
else
achout=$1
fi
# Set session cookies
rm /tmp/fedc.txt 2>/dev/null
curl -s -c /tmp/fedc.txt \
https://www.frbservices.org/EPaymentsDirectory/FedACHdir.txt -o /dev/null
# Submit agreement
curl -s -b /tmp/fedc.txt -c /tmp/fedc.txt -d "agreementValue=Agree" \
-H "Referer:https://www.frbservices.org/EPaymentsDirectory/agreement.html" \
https://www.frbservices.org/EPaymentsDirectory/submitAgreement -o /dev/null
#read FedACH catalog
curl -b /tmp/fedc.txt \
https://www.frbservices.org/EPaymentsDirectory/FedACHdir.txt -o $achout
# Cleanup cookies
rm /tmp/fedc.txt 2>/dev/null
If you find this useful: enjoy and have fun.
If you find this useful: enjoy and have fun.
