Make you loading faster with parallel processing
After know how to unloading data using OCIULDR, it time to Load your data to the original table. Just like the last post this script using list of table as the input and the data using pattern like [table_name].dat just like output from the script before.
Sample list of table:
filename : list_of_tables.txt
content :
table_a
table_b
Using pattern above the data will be like this
table_a.dat
table_b.dat
and the scripts are :
1. run
#/bin/bash
if [ $# -lt 6 ]
then
echo "Please use arguments, $0 [USERNAME] [PASSWORD] [SID] [LIST OF TABLE] [THREADNAME] [MAX THREAD]"
exit 1
fi
#[ Configure your directory ]
CTLDIR=....
BADDIR=....
LOGDIR=....
INDIR=....
USERNAME=$1
PASSWORD=$2
SID=$3
sList=$4
THREADNAME=$5
max_thread=$6
THREADID=0
##########MAIN PROGRAM
######################################################################################################
echo "Start loading process : `date`"
for FILENAME in `cat $sList`
do
##identified any threads
sFlag=0
while [ $sFlag -eq 0 ]
do
NEXT_PROC=`ls ${THREADNAME}_*.lock | wc -l`
if [ $NEXT_PROC -le $max_thread ]
then
sFlag=1
else
sleep 15
fi
done
#running others threads
touch ${THREADNAME}_${THREADID}.lock
DATAFILE=${FILENAME}.dat
if [ -e ${INDIR}/${DATAFILE} ]
then
echo "Loading.......${DATAFILE}"
. loader $USERNAME $PASSWORD $SID ${FILENAME} ${THREADNAME}_${THREADID} &
else
echo "File $DATAFILE tidak ditemukan." >> RUNNER_LOADING.log
echo "Releasing lock"
rm ${THREADNAME}_${THREADID}.lock
fi
let "THREADID+=1"
done
2. loader
#/bin/bash
if [ $# -lt 5 ]
then
echo "Please use arguments, $0 [USERNAME] [PASSWORD] [SID] [FILENAME] [THREAD NAME]"
exit 1
fi
USERNAME=$1
PASSWORD=$2
SID=$3
FILENAME=$4
THREADNAME=$5
###THIS IS MANDATORY
##ORACLE LIBRARY
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
########MAIN PROGRAM
CTLFILE=${CTLDIR}/${FILENAME}._sqlldr.ctl
BADFILE=${BADDIR}/${FILENAME}.bad
LOGFILE=${LOGDIR}/${FILENAME}.log
DATAFILE=${INDIR}/${FILENAME}.dat
echo "Start time : `date` $THREADNAME"
echo "Loading CTLFILE : ${CTLFILE}"
echo "Loading FILENAME : ${DATAFILE}"
# Load data into table
sqlldr ${USERNAME}/${PASSWORD}@${SID} control=${CTLFILE} data=${DATAFILE} log=${LOGFILE} bad=${BADFILE} direct=true parallel=yes errors=1000000
if [ -a ${THREADNAME}.lock ]
then
rm ${THREADNAME}.lock
fi
echo "End time : `date`"
echo ""
##END PROGRAM
Implementation:
Don’t forget to configure this part with your own
#[ Configure your directory ]
CTLDIR=….
BADDIR=….
LOGDIR=….
INDIR=….
after that you can run with command
./run [USERNAME] [PASSWORD] [SID] [LIST OF TABLE] [THREADNAME] [MAX THREAD]
Good Luck
SRO
About this entry
You’re currently reading “Make you loading faster with parallel processing,” an entry on Zhobur's
- Published:
- May 5, 2009 / 9:10 am
- Category:
- Oracle PL/SQL, Shell
- Tags:
2 Comments
Jump to comment form | comment rss [?] | trackback uri [?]