Make you loading faster with parallel processing

After know how to unloading data using OCIULDR, it time to Load your data to the original table. Just like the last post this script using list of table as the input and the data using pattern like [table_name].dat just like output from the script before.
Sample list of table:
filename : list_of_tables.txt
content :
table_a
table_b

Using pattern above the data will be like this
table_a.dat
table_b.dat

and the scripts are :
1. run

#/bin/bash

if [ $# -lt 6 ]
then
   echo "Please use arguments, $0 [USERNAME] [PASSWORD] [SID] [LIST OF TABLE] [THREADNAME] [MAX THREAD]"
   exit 1
fi

#[ Configure your directory ] 
CTLDIR=....
BADDIR=....
LOGDIR=....
INDIR=....

USERNAME=$1
PASSWORD=$2
SID=$3
sList=$4
THREADNAME=$5
max_thread=$6

THREADID=0
##########MAIN PROGRAM
######################################################################################################
echo "Start loading process : `date`"

for FILENAME in `cat $sList`
do
   ##identified any threads
   sFlag=0
   while [ $sFlag -eq 0 ]
   do
      NEXT_PROC=`ls ${THREADNAME}_*.lock | wc -l`
      if [ $NEXT_PROC -le $max_thread ]
      then
         sFlag=1
      else
         sleep 15
      fi
   done

   #running others threads
   touch ${THREADNAME}_${THREADID}.lock
   DATAFILE=${FILENAME}.dat

   if [ -e ${INDIR}/${DATAFILE} ]
   then

      echo "Loading.......${DATAFILE}"
      . loader $USERNAME $PASSWORD $SID ${FILENAME} ${THREADNAME}_${THREADID} &

   else
      echo "File $DATAFILE tidak ditemukan." >> RUNNER_LOADING.log
      echo "Releasing lock"
      rm ${THREADNAME}_${THREADID}.lock
   fi

   let "THREADID+=1"

done

2. loader

#/bin/bash

if [ $# -lt 5 ]
then
   echo "Please use arguments, $0 [USERNAME] [PASSWORD] [SID] [FILENAME] [THREAD NAME]"
   exit 1
fi

USERNAME=$1
PASSWORD=$2
SID=$3
FILENAME=$4
THREADNAME=$5

###THIS IS MANDATORY
##ORACLE LIBRARY
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH

########MAIN PROGRAM
CTLFILE=${CTLDIR}/${FILENAME}._sqlldr.ctl
BADFILE=${BADDIR}/${FILENAME}.bad
LOGFILE=${LOGDIR}/${FILENAME}.log
DATAFILE=${INDIR}/${FILENAME}.dat


echo "Start time : `date` $THREADNAME" 
echo "Loading CTLFILE : ${CTLFILE}"
echo "Loading FILENAME : ${DATAFILE}"


# Load data into table
sqlldr ${USERNAME}/${PASSWORD}@${SID} control=${CTLFILE} data=${DATAFILE} log=${LOGFILE} bad=${BADFILE} direct=true parallel=yes errors=1000000


if [ -a ${THREADNAME}.lock ]
then
   rm ${THREADNAME}.lock
fi

echo "End time : `date`"
echo ""
##END PROGRAM

Implementation:
Don’t forget to configure this part with your own
#[ Configure your directory ]
CTLDIR=….
BADDIR=….
LOGDIR=….
INDIR=….

after that you can run with command
./run [USERNAME] [PASSWORD] [SID] [LIST OF TABLE] [THREADNAME] [MAX THREAD]

Good Luck
SRO


About this entry