Hi, First off, let me admit, im a java newbie. I've been using java for about 6 months now, although im a fast learner, some things still elude me.
I've searched these forums and google high and low for an answer to my question before signing up and posting here, so i'm kind of at my wits end.
Let me explain my specs so noone blames lack of power for this issue.
System specs:
Linux Cent OS 4 2.6.19 SMP
JRE 1.6
MySQL 4.2.2 ICC Compiled Enterprise Edition
4x HyperThreaded Xeon 1.9Ghz cpus (Displays as 8)
16GB ECC Double-Parity Double-Buffered Memory. (Rediculous IBM proprietary memory)
Problem:
I first wrote this application in PHP as it my stong suit (that should tell you plenty). But the PHP CLI was eating up way to much CPU power/MySQL bandwith to execute this script, and taking way to long, so i started re-writing it in C++, well when my feeble C++ powers failed me, i went to Java. I know more Java then C++ but i picked C first because i thought it would be the fastest. Java, however, is slower then PHP when i write the identical logic, i dont know if i should blame Java or my assumption that i can copy logic from one language to another.
Here is the details:
I have 1 MySQL database and 2 Tables im pulling from, This mysql database is optimized by the people from MySQL, so i know its not the DB, Server, or Tables. 1 table (bfpax) has around 45k rows (45000), the second table (bfbdet) has around 100k rows (100000). What this program has to do, its objective, is:
1. Pull the indexed ID from table 1;
2. With that ID, pull other data from table 2 and table 1 for each ID;
3. Compile that data together for each ID
4. Insert compiled data into a HEAP table for fast access for each ID.
In PHP (pseudocode):
SELECT ID FROM bfpax;
Loop:
Foreach id pull various data pieces i.e. name, client code, address, zip, etc.
String format name: "A-GARY/COLE" becomes "Gary/Cole"
String format hotel: "ACUNMIP" becomes CUNMIP
Insert all formatted/fixed data into heap table
End loop.
In Java (pseudocode):
SELECT ID FROM cpax;
Loop:
Create object with ID as constructor variable (see below for object declaration) and add to an arrayList;
End Loop;
Create 6 threads to execute each of the objects methods for data collection/formatting
Start each thread using inner classes to loop through the object arraylist and execute dedicated methods per thread.
Thread 1: Pull air data
Thread 2: Pull destination
Thread 3: Pull hotel
Thread 4: Pull gateway
Thread 5: Pull the rest (price, dates, etc)
Thread 6: Start executing the update method
The above kind of failed, so i reduced the complexity by nuking the whole thread idea and now i just have a simple function that loops through the arraylist of objects and executes each method in order.
Booking Object:
public class Booking extends ReportingConstants{
private ArrayList<String> myData = new ArrayList<String>();
private int myBookNum;
private static Connection myCon;
public static int numBookings;
public int updateCount = 0;
public boolean isFinished = false;
public Booking(int booking_number, Connection conn) {
if (booking_number > 9999) {
myBookNum = booking_number;
}
numBookings++;
myCon = conn;
//Run loop to make sure myData has the proper spaces for insert, ensureCapacity didnt work.
for (int i = 0; i < 15; i++) {
myData.add("");
}
myData.set(_BOOKNUM,String.valueOf(myBookNum));
if (Integer.valueOf(_ARGS[_DEBUG][1]) > 0) {
System.out.println("Initialized booking number "+myBookNum+" count "+numBookings);
}
}
public void getAir() {
ResultSet res;
ArrayList<String> total_air = new ArrayList<String>();
String airlines = "";
String query = "SELECT operator_id FROM EDITED.bfbdet WHERE record_type = 'A' AND item_desc LIKE '-%' AND booking_number = "+myBookNum+" AND operator_id IS NOT NULL";
if (Integer.valueOf(_ARGS[_DEBUG][1]) > 1) {
System.out.println("Starting Air: "+query);
}
res = MySQL.sqlQuery(query,myCon,5);
try {
while(res.next()) {
if (res.getString(1).length() > 1) {
String id = res.getString(1).substring(0,2);
if (!total_air.contains(id)) {
airlines += id+",";
}
total_air.add(id);
}
}
res.close();
MySQL.close();
} catch (Exception ex) {
ex.printStackTrace();
}
myData.set(_AIR,airlines);
updateCount++;
}
public void getDest() {
String query = "SELECT booking_code FROM EDITED.bfpax WHERE booking_number = "+myBookNum+" LIMIT 1";
ResultSet res;
String isSv ="Y";
String dest;
if (Integer.valueOf(_ARGS[_DEBUG][1]) > 1) {
System.out.println("Starting Dest: "+query);
}
res = MySQL.sqlQuery(query,myCon,1);
try {
while (res.next()) {
dest = res.getString(1).substring(1,4);
if (dest != "FRE") {
if (dest == "GYY") {
String realDest = res.getString(1).substring(4,2);
if (realDest == "GY") {
dest = "GYY";
} else if (realDest == "MC") {
dest = "MCO";
} else if (realDest == "FL") {
dest = "FLL";
} else if (realDest == "IW") {
dest = "IWA";
} else if (realDest == "PI") {
dest = "PIE";
} else if (realDest == "LA") {
dest = "LAS";
} else {
dest = "GYY";
}
} else {
isSv = "N";
}
myData.set(_DEST,dest);
myData.set(_SV,isSv);
updateCount++;
}
}
res.close();
MySQL.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
public void getGateway() {
String query = "SELECT item_desc FROM EDITED.bfbdet WHERE (booking_number = '"+myBookNum+"' OR booking_number = ' "+myBookNum+"') AND item_desc LIKE '-%' ORDER BY booking_suffix ASC LIMIT 1";
if (Integer.valueOf(_ARGS[_DEBUG][1]) > 1) {
System.out.println("Starting GW: "+query);
}
ResultSet res = MySQL.sqlQuery(query,myCon,1);
String gw;
try {
while (res.next()) {
gw = res.getString(1).substring(1,3);
myData.set(_GW,gw);
}
updateCount++;
res.close();
MySQL.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
public void getHotel() {
String query = "SELECT operator_id FROM EDITED.bfbdet WHERE record_type = 'H' AND (booking_number = '"+myBookNum+"' OR booking_number = ' "+myBookNum+"') LIMIT 1";
if (Integer.valueOf(_ARGS[_DEBUG][1]) > 1) {
System.out.println("Starting Hotel: "+query);
}
ResultSet res = MySQL.sqlQuery(query,myCon,1);
String hotel = "";
try {
while (res.next()) {
hotel = res.getString(1).substring(0,6);
if (myData.get(_DEST) == "") {
myData.set(_DEST,res.getString(1).substring(0,3));
updateCount++;
}
}
myData.set(_HOTEL,hotel);
updateCount++;
res.close();
MySQL.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
public void getRest() {
String query = "SELECT client_code, passenger1_name, agentid, booked_date, dep_date, total_price, total_received, total_commission, number_pax FROM EDITED.bfpax WHERE booking_number = "+myBookNum+" LIMIT 1";
if (Integer.valueOf(_ARGS[_DEBUG][1]) > 1) {
System.out.println("Starting Rest: "+query);
}
ResultSet res = MySQL.sqlQuery(query,myCon,1);
try {
while (res.next()) {
myData.set(_AGENCY,res.getString(1));
if (res.getString(3) != "null") {
myData.set(_AGENT,res.getString(3));
} else {
myData.set(_AGENT,"");
}
myData.set(_PAXNAME,res.getString(2).replace("'",""));
myData.set(_BKDATE,String.valueOf(res.getDate(4)));
myData.set(_DEPDATE,String.valueOf(res.getDate(5)));
myData.set(_TPRICE,String.valueOf(res.getDouble(6)));
myData.set(_TRECV,String.valueOf(res.getDouble(7)));
myData.set(_TCOM,String.valueOf(res.getDouble(8)));
myData.set(_NUMPAX,String.valueOf(res.getInt(9)).trim());
//System.out.println("NUMPAX: |"+myData.get(_NUMPAX)+"|");
}
updateCount++;
res.close();
MySQL.close();
} catch(Exception ex) {
ex.printStackTrace();
}
}
public void storeData() {
if (!isFinished) {
String query = "INSERT INTO "+tmpTable+" (`booking_number`, `destination`, `gateway`, `airline`, `hotel`, `agency`, `agent`, `booked_date`, `dep_date`, `total_price`, `total_received`, `total_commission`, `number_pax`, `passenger_name`, `is_skyvalue`) VALUES('"+myData.get(0)+"','"+myData.get(1)+"','"+myData.get(2)+"','"+myData.get(3)+"','"+myData.get(4)+"','"+myData.get(5)+"','"+myData.get(6)+"','"+myData.get(7)+"','"+myData.get(8)+"','"+myData.get(9)+"','"+myData.get(10)+"','"+myData.get(11)+"','"+myData.get(12)+"','"+myData.get(13)+"','"+myData.get(14)+"')";
if (Integer.valueOf(_ARGS[_DEBUG][1]) > 1) {
System.out.println(query);
}
MySQL.sqlExec(query,myCon);
isFinished = true;
if (Integer.valueOf(_ARGS[_DEBUG][1]) > 0) {
System.out.println("Booking number "+myBookNum+" is finished!");
}
} else {
//System.out.println("Not fully populated!");
}
}
}
Please dont laugh too hard heh, i know my code is sloppy and probably not optimized at all, i dont pretend to be a hardcore java guy, but i would love to learn. Im also posting below my static MySQL class so you can see what happens when i call sqlQuery();
public abstract class MySQL {
private static ResultSet res;
private static Statement stmt;
public static ResultSet sqlQuery(String query,Connection con, int limit) {
try {
stmt = con.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(limit);
res = stmt.executeQuery(query);
} catch(Exception ex) {
ex.printStackTrace();
res = null;
System.out.println("Cant Query!: "+query);
}
return res;
}
public static void sqlExec(String query,Connection con) {
Statement stmt;
try {
stmt = con.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_READ_ONLY);
stmt.executeUpdate(query);
stmt.close();
} catch(Exception ex) {
ex.printStackTrace();
System.out.println("Cant Query!: "+query);
}
}
public static void close() {
try {
stmt.close();
res.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
If you notice, i require the connection to be passed when executing a query, i do this because even if i made a static connection state in the MySQL class, for some reason, it still gets duplicated every time its called. This way, i create 1 connection with the calling static main class, and pass that same conn back and forth through the whole script. I know in PHP this is the best way to do it, in Java it may require many different connections.
By the way, this is an update that runs once per hour, it does not have to worry about users or anything like that, it gets executed by cron on the hour.
PHP does this job of updating about 40000 entries in 9-12 minutes. for an average of 0.0135 seconds per entry, Java rolls around at 0.06 seconds per query, 6x slower, this puts execution time at about 40 minutes, obviously this is an issue if its supposed to run every hour. The java app does not tax the system, on top/htop it does not even make it to the first page so its not resources, ive looked at the verbose gc info, and i didnt see anything to clue me in there either.
I'm sure this is a simple error and my lack of knowledge with Java is what's holding me back, but if i cant figure this out ill have to go back to struggling with C++.
Thanks in advance, and im sorry for the horribly long post.
Thanks,
Dave