SphinxQL over JDBC

This is an extension of my earlier blog post about performing free text search with Sphinx (v2.1.6+). This time we are going to access the index via JDBC as opposed to Sphinx API and perform some basic queries to fetch indexed data.

Add following entries to searchd section inside sphinx.conf
listen=9306:mysql41
mysql_version_string=5.0.37

Once done start the sphinx daemon (searchd) so that the search engine can be accessed from external applications on port 9306. Next just get the JDBC connection (no need to specify database details and credentials) as shown below and you are ready to rock.
Class.forName("com.mysql.jdbc.Driver");
con = DriverManager
 .getConnection("jdbc:mysql://localhost:9306");
statement = con.createStatement();
// perform wild search on last names
resultSet = statement
 .executeQuery("select * from addressBookIndex 
where MATCH ('gogna | sharma')"); 

Note:
  • 1. The attributes specified in the sphinx.conf file (e.g. sql_attr_string, sql_attr_uint etc) can not be used with MATCH though they can be part of WHERE clause.
  • 2. Given that now String attributes are supported in order to avoid making a database call define key columns as attributes as well as fields in sql_query.
  • 3. Results from Delta Indexes can be handled using union.
  • 4. Explore RT indexes for real time update scenarios.
For more information about the Sphinx QL syntax visit here.
Saturday, March 15, 2014
Posted by Munish Gogna

Apache Velocity Template to generate CSV (or any other format) file

This post is simple one to generate CSV file from some Source using Velocity Template and to verify the new design for the Blog (Metro Blue). The code (just a running stuff and nothing fancy) provided is simple and doesn't need any explanation and is dedicated to a dear friend Sachin Shah.






Java Class
package com.gognamunish.test;

import org.apache.velocity.app.Velocity;
import org.apache.velocity.Template;
import org.apache.velocity.VelocityContext;

import java.io.File;
import java.io.FileWriter;
import java.io.StringWriter;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class CSVFeedGeneration {

  static DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");

  static DataFeed dataFeed;
 static {
  // prepare some static data
  dataFeed = new DataFeed();
  List<Map<String, Object>> dataList = new ArrayList<Map<String, Object>>();
  Map<String, Object> data = new HashMap<String, Object>();
  data.put("TransactionId", 1);
  data.put("Amount", new Integer(2312));
  data.put("Currency", "SGD");
  data.put("ExecutionDate", dateFormat.format(new Date()));
  dataList.add(data);
  data = new HashMap<String, Object>();
  data.put("TransactionId", 2);
  data.put("Amount", new Integer(1212));
  data.put("Currency", "INR");
  data.put("ExecutionDate", dateFormat.format(new Date()));
  data.put("Status", "Active");
  dataList.add(data);

   dataFeed.getFeedData().add(dataList);

  }

  public static void main(String[] args) throws Exception {
  VelocityContext context = new VelocityContext();
  FileWriter writer = new FileWriter(new File("feed.csv"));
  
  // inject the objects to be used in template
  context.put("feed", dataFeed);
  context.put("DELIMETER", ",");
  context.put("NEWLINE", "\n");

  Velocity.init();
  Template template = Velocity.getTemplate("csvfeed.vm");
  template.merge(context, writer);
  writer.flush();
  writer.close();

  }

}
Template File (csvfeed.vm)
TransactionId${DELIMETER}Amount${DELIMETER}ExecutionDate ${DELIMETER}Currency${DELIMETER}Status$NEWLINE
#foreach( $rowSet in $feed.feedData)
#foreach( $row in $rowSet )
$!row.get("TransactionId")$DELIMETER$!row.get("Amount") $DELIMETER$!row.get("ExecutionDate")$DELIMETER $!row.get("Currency")$DELIMETER$!row.get("Status")$NEWLINE
#end
#end
Output File (feed.csv)
TransactionId,Amount,ExecutionDate,Currency,Status
1,2312,2013-08-18T14:29:35,SGD,
2,1212,2013-08-18T14:29:35,INR,Active
The main advantage of this approach is that you can change the data format any time without changing or touching any Java code.
POM Dependency
<dependency>
   <groupId>org.apache.velocity</groupId>
   <artifactId>velocity</artifactId>
   <version>1.7</version>
  </dependency>
In case you are interested in exploring more, reach out here http://velocity.apache.org/engine/devel/index.html
Sunday, August 18, 2013
Posted by Munish Gogna

Configuring IBM MQ in JBoss


In today's post I will explain how to install the WebSphere MQ resource adapter on JBoss.

Installing WebSphere MQ resource adapter:
To install WebSphere MQ resource adapter, copy the file wmq.jmsra.rar (from installation directory) to the server's deploy directory, for example <install path>/server/default/deploy The resource adapter will be automatically picked up by the server.

Installing the WebSphere MQ Extended Transactional Client:
The WebSphere MQ Extended transactional client lets you use XA distributed transactions with CLIENT mode connections to a WebSphere MQ queue manager. To install it on JBoss, copy the file com.ibm.mqetclient.jar ( from installation directory ) to the server lib directory, for example <install path>/server/default/lib.

Configuring the resource adapter for your application:
The WebSphere MQ resource adapter lets you define a number of global properties. Resource definitions for outbound JCA flows are defined in a JBoss -ds.xml file named wmq.jmsra-ds.xml. The outline form of this file:

  
  
  
  



As can be seen above JCA connection factories are defined in the <tx-connection-factory> element and JMS queues and topics are defined as JCA administered object mbeans.

A complete example for our sample application defining two queues - inbound and outbound is given below:

    
        jms/MyAppConnectionFactory
        wmq.jmsra.rar
        true
        javax.jms.QueueConnectionFactory
        8
        36
        ${channel}
        ${hostName}
        1414
        ${queueManager}
        CLIENT
        munish
        JmsXARealm
        
    
    
  jms/IncomingQueue
  jboss.jca:name='wmq.jmsra.rar',service=RARDeployment
  javax.jms.Queue
  
   baseQueueManagerName=${queueManager}
   baseQueueName=MY.APP.INBOUND.QUEUE
    
    
    
        jms/OutgoingQueue
        jboss.jca:name='wmq.jmsra.rar',service=RARDeployment
        javax.jms.Queue
        
            baseQueueManagerName=${queueManager}
            baseQueueName=MY.APP.OUTBOUND.QUEUE
        
    



The place holder properties are defined in the JBoss start script as:
-Dchannel=MQTEST -DhostName=localhost -DqueueManager=TST_MGR

Configuring inbound message delivery:
Message delivery to a Message-Driven Enterprise JavaBean (MDB) is configured using a JBoss-specific file named jboss.xml, located in the MDB .ear file. The contents of this file will vary, but will be of the form:

jboss.xml:

 
  
   MyMessageBean
    
     
                     
                             DestinationType
                             javax.jms.Queue
                     
                     
                             destination
                             MY.APP.INBOUND.QUEUE
                     
                     
                             channel
                             ${channel}
                     
                     
                             hostName
                             ${hostName}
                     
                     
                             port
                             1414
                     
                     
                             queueManager
                             ${queueManager}
                     
                     
                             transportType
                             CLIENT
                     
                     
                             username
                             munish
                     
              
               
           wmq.jmsra.rar
     
 


The DestinationType and Name properties must be specified. Other properties are optional, and if omitted, take on their default values.

A simple MDB definition for the inbound message is shown below:
/**
 * This message driven bean is used to collect asynchronously received messages.
 *
 * @author Munish Gogna
 */
@MessageDriven(mappedName = "jms/IncomingQueue")
public final class MyMessageBean implements javax.jms.MessageListener {
  private static final org.apache.log4j.Logger LOGGER = org.apache.log4j.Logger.getLogger(MyMessageBean.class);
  public void onMessage(final javax.jms.Message message) {
    LOGGER.info("Received message: " + message);
    // TODO - do processing with the message here
  }
}


A plain java class to send message to the output queue is shown below.
/**
 * This class is used to send text messages to a queue.
 *
 * @author Munish Gogna
 */
public final class MessageSender {
  private static final org.apache.log4j.Logger LOGGER = org.apache.log4j.Logger.getLogger(MessageSender.class);
  /**
   * A coarse-grained method, nothing fancy. In actual applications please re factor the code and
   * handle the exceptions properly.
   *
   * @throws Exception
   */
  public void send(final String message) throws Exception {
    if (message == null) {
      throw new IllegalArgumentException("Parameter message cannot be null");
    }
    javax.jms.QueueConnection connection = null;
    javax.jms.Session session = null;
    try {
      javax.naming.Context fContext = new javax.naming.InitialContext();
      javax.jms.QueueConnectionFactory fConnectionFactory = (javax.jms.QueueConnectionFactory) fContext
          .lookup("java:jms/MyAppConnectionFactory");
      connection = fConnectionFactory.createQueueConnection();
      session = connection.createSession(false, javax.jms.Session.AUTO_ACKNOWLEDGE);
      final javax.naming.Context jndiContext = new javax.naming.InitialContext();
      final javax.jms.Destination destination = (javax.jms.Destination) jndiContext.lookup("jms/OutgoingQueue");
      final javax.jms.MessageProducer messageProducer = session.createProducer(destination);
      final javax.jms.TextMessage textMessage = session.createTextMessage(message);
      messageProducer.send(textMessage);
      messageProducer.close();
    } catch (Exception nex) {
      throw nex;
    } finally {
      if (session != null) {
        try {
          session.close();
        } catch (JMSException e) {
          LOGGER.error("Failed to close JMS session", e);
        }
      }
      if (connection != null) {
        try {
          connection.close();
        } catch (JMSException e) {
          LOGGER.error("Failed to close JMS connection", e);
        }
      }
    }
  }
}
A utilty called RFHUTILC.EXE can be used to connect to remote queues.

That's all.
Monday, June 4, 2012
Posted by Munish Gogna

Saving development time with local EntityManager in JPA

In almost all real world projects (non Spring + We are not considering Arquillian or any similar framework) where you are using JPA (Hibernate implementation), the data source binding happens by JNDI and the services are provided by EJBs, which means to test your entities for any CRUD operation you have to deploy the bundle (ear, ejb or war) into the application server which adds time to your development efforts each time you make any change. So in this post we will see how we can temporarily inject entity manager into your business class so that you can test all the operations you want to perform on entities without deploying the application or restarting server quite often.

persistence.xml: The configuration for entity managers both inside an application server and in a standalone application reside in a persistence archive. In the next section we focus on standalone configuration.

 
  org.hibernate.ejb.HibernatePersistence
  com.gognamunish.test.Entity
  
   
   
   
   
   
  
 

Please Note:
  • The database dialect we have chosen is MySQL for obvious reasons.
  • Persistence unit has name test_unit and the type is RESOURCE_LOCAL and not JTA.
  • com.gognamunish.test.Entity is the entity which we want to test later.
  • As we are running without a JNDI available datasource, we must specify JDBC connections with Hibernate specific properties
Next in your business class you can get reference to the entity manager as follow:
EntityManagerFactory factory ;
factory = Persistence.createEntityManagerFactory("test_unit");
EntityManager manager = factory.createEntityManager();
That's all you need , after obtaining the reference focus only on your entity and the operation you want to perform on it.
Let's try one sample example:
/* Sample Entity class */
@javax.persistence.Entity
@Table(name = "Entity")
public class Entity {
 @Id
 private Long id;
 @Column
 private String value;
}
package com.gognamunish.test;

import java.util.List;

import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;

/**
 * Sample Test - nothing fancy.
 * 
 * @author Munish Gogna
 * 
 */
public class LocalEntityManager {
 static EntityManager manager;
 static {
  EntityManagerFactory factory = Persistence
    .createEntityManagerFactory("test_unit");
  manager = factory.createEntityManager();
 }

 public static void main(String[] args) {

  System.out.println("Before Entities: " + getEntities());

  manager.getTransaction().begin();
  Entity entity = new Entity();
  entity.setId(1l);
  entity.setValue("my first value");
  saveOrUpdate(entity);
  manager.getTransaction().commit();

  System.out.println("After Entities: " + getEntities());

 }

 private static Entity saveOrUpdate(Entity entity) {
  return manager.merge(entity);
 }

 @SuppressWarnings("unchecked")
 private static List<Entity> getEntities() {
  return manager.createQuery("from Entity").getResultList();
 }
}

LOGS:
Hibernate: select entity0_.id as id0_, entity0_.value as value0_ from Entity entity0_
Before Entities: []
Hibernate: select entity0_.id as id0_0_, entity0_.value as value0_0_ from Entity entity0_ where entity0_.id=?
Hibernate: insert into Entity (value, id) values (?, ?)
Hibernate: select entity0_.id as id0_, entity0_.value as value0_ from Entity entity0_
After Entities: [Id:1, Value:my first value]
Sunday, April 29, 2012
Posted by Munish Gogna

Reverse a String in Java

One of the frequently asked questions in interviews, let's try few ways available in JDK to reverse a string:





import java.util.Stack;
public class StringReverse {

 public static String reverse1(String s) {
  int length = s.length();
  if (length <= 1)
   return s;
  String left = s.substring(0, length / 2);
  String right = s.substring(length / 2, length);
  return reverse1(right) + reverse1(left);
 }

 public static String reverse2(String s) {
  int length = s.length();
  String reverse = "";
  for (int i = 0; i < length; i++)
   reverse = s.charAt(i) + reverse;
  return reverse;
 }

 public static String reverse3(String s) {
  char[] array = s.toCharArray();
  String reverse = "";
  for (int i = array.length - 1; i >= 0; i--)
   reverse += array[i];

  return reverse;
 }

 public static String reverse4(String s) {
  return new StringBuffer(s).reverse().toString();
 }

 public static String reverse5(String orig) {
  char[] s = orig.toCharArray();
  int n = s.length - 1;
  int halfLength = n / 2;
  for (int i = 0; i <= halfLength; i++) {
   char temp = s[i];
   s[i] = s[n - i];
   s[n - i] = temp;
  }
  return new String(s);
 }

 public static String reverse6(String s) {

  char[] str = s.toCharArray();

  int begin = 0;
  int end = s.length() - 1;

  while (begin < end) {
   str[begin] = (char) (str[begin] ^ str[end]);
   str[end] = (char) (str[begin] ^ str[end]);
   str[begin] = (char) (str[end] ^ str[begin]);
   begin++;
   end--;
  }

  return new String(str);
 }

 public static String reverse7(String s) {
  char[] str = s.toCharArray();
  Stack<Character> stack = new Stack<Character>();
  for (int i = 0; i < str.length; i++)
   stack.push(str[i]);

  String reversed = "";
  for (int i = 0; i < str.length; i++)
   reversed += stack.pop();

  return reversed;
 }

}


Feel free to suggest other ways if you have.
Sunday, March 4, 2012
Posted by Munish Gogna

Merging PDFs while retaining the fill able forms inside them

My previous post was about merging the PDF files into single file using PDFBox, the limitation of that approach was if you had fill able form inside any PDF file it was lost, to over come that issue here is an updated version of the same post using iText library.







import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;

import com.lowagie.text.pdf.AcroFields;
import com.lowagie.text.pdf.PdfCopyFields;
import com.lowagie.text.pdf.PdfReader;

/**
 * Merges the PDF files while retaining the fillable forms within them. It also
 * renames form field (to avoid conflicts) which can have an impact if form is
 * submitted to server but for printing purpose should be OK.
 * 
 * @author Munish Gogna
 * 
 */
public class PDFMerge {
 private static int counter = 0;

 public static void merge(List<InputStream> sourcePDFs,
   OutputStream targetPDF) throws Exception {
  if (sourcePDFs != null && targetPDF != null) {
   PdfCopyFields copy = new PdfCopyFields(targetPDF);

   for (InputStream stream : sourcePDFs) {
    PdfReader pdfReader = new PdfReader(stream);
    renameFields(pdfReader.getAcroFields());
    copy.addDocument(pdfReader);
    stream.close();
   }
   copy.close();
  }
 }

 private static void renameFields(AcroFields fields) {
  Set<String> fieldNames = fields.getFields().keySet();
  String prepend = String.format("_%d.", counter++);

  for (String fieldName : fieldNames) {
   fields.renameField(fieldName, prepend + fieldName);
  }
 }

 public static void main(String[] args) {
  try {
   List<InputStream> pdfs = new ArrayList<InputStream>();
   pdfs
     .add(new URL(
       "http://thewebjockeys.com/TheWebJockeys/Fillable_PDF_Sample_from_TheWebJockeys_vC5.pdf")
       .openStream());
   pdfs
     .add(new URL(
       "http://help.adobe.com/en_US/Acrobat/9.0/Samples/interactiveform_enabled.pdf")
       .openStream());
   OutputStream output = new FileOutputStream("merged.pdf");

   merge(pdfs, output);

  } catch (Exception e) {
   e.printStackTrace();
  }
 }

}

The iText jar can be downloaded here

Thanks.
Sunday, January 15, 2012
Posted by Munish Gogna

Merging PDF files in Java

I was just looking for some way to merge PDFs generated from different sources to one final deck. iText library was one obvious choice but when I looked into PDFBox library I was stunned with the simplicity and ease with which PDFs can be merged. I tried merging three different PDF files and to my surprise it took less than 15 lines of code as shown below:



package com.gognamunish.test;

import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import org.apache.pdfbox.util.PDFMergerUtility;

/**
 * Merges pdf files into one final pdf.
 * 
 * @author Munish Gogna
 */
public class PDFMerge {

 public static void main(String[] args) throws Exception {
  // Get the byte streams from any source (maintain order)
  List<InputStream> sourcePDFs = new ArrayList<InputStream>();
  sourcePDFs.add(new FileInputStream(new File("pdf1.pdf")));
  sourcePDFs.add(new FileInputStream(new File("pdf2.pdf")));
  sourcePDFs.add(new FileInputStream(new File("pdf3.pdf")));

  // initialize the Merger utility and add pdfs to be merged
  PDFMergerUtility mergerUtility = new PDFMergerUtility();
  mergerUtility.addSources(sourcePDFs);
  // set the destination pdf name and merge input pdfs
  mergerUtility.setDestinationFileName("merged.pdf");
  mergerUtility.mergeDocuments();
 }
}

The library is rich and provides many options out of the box that can be useful when it comes to playing with PDFs, worth a try !!!

http://pdfbox.apache.org/index.html
 cheers, Munish Gogna
Saturday, November 5, 2011
Posted by Munish Gogna

Windows 8 - carbon copy of Apple's business model...all in ones

I am personally not impressed with what Windows 8 has to offer. In short here is the Summary:
  • Responsive start and shut down time - more like Apple devices.
  • Slider UIs copied from iPhone and similar mobile devices.
  • Metro UI - It's boring over time and just gets in the way.
  • Integrated App Store, create an app and sell it for 99 paise :) Finally Microsoft realized the power of developers.
  • Notifications concept copied from Facebook. Blue Screen of Death gets makeover for Windows 8, I am not sure from wehere Microsoft copied “Blue Screen of Death”, any ideas guys?

HOME SCREEN

DESKTOP

CONTROL PANEL

ERRORS
Sunday, September 18, 2011
Posted by Munish Gogna

Performance Analysis for Enterprise Java Applications

Users come to you and shout "Why Trades page is taking too much time to load?" Now you may say because I call Trade Service so it might be taking time to return the response in timely fashion. You approach developer of the Service and ask - Can you please tell how much time is being taken for getTrades(userId) to return trades from database for XYZ user? Developer will say Yes I can - let me check you in the logs as I have put System.currentTimeMillis(); statements in my code as shown below:

public static List<Trade> getTrades(String userId)
			throws TradeServiceException {
		long start = System.currentTimeMillis();
		// separate component running in separate JVM
		authenticate(userId);
		// separate component running in separate JVM
		authorize(userId);

		List<Trade> trades = TradeDao.getTrades(userId);
		long end = System.currentTimeMillis();
		System.out.println("Method took " + ((end - start) / 1000) + " seconds");
		return trades;

	}
As you can see in the code snippet that this method relies on external components (common to all services) for actual task. So answering the question "Why Trades page is slow?" requires investigating multiple components and execution paths, and it requires detailed performance data from all of the application components used. It may be due to poor performance of component that perform authorization for user.

The developer here forgot that the total time printed in logs is basically sum of time taken for authentication, authorization and actual database call. So he may change the code to add few more System.currentTimeMillis(); statements around authenticate and authorize methods to come up with the individual timings for each component (He feels happy and relaxed).

Later on, though, architects find they want more information, such as aggregated performance statistics like mean, minimum, maximum, standard deviation and transactions per second over a set time span. They would like graphs of these values in real time to detect problems on a running server, or they would like to expose performance metrics through JMX so that monitors can be set up to alert when performance degrades. In addition, they want their timing statements to work with the common logging frameworks like log4j (so that they can be enabled and disabled at run time).

Can someone save this developer from pain? The answer is YES - It's Perf4J.
"Perf4J is to System.currentTimeMillis() as log4j is to System.out.println()"

Perf4J provides above features and more:
  • A simple stop watch mechanism for succinct timing statements.
  • A command line tool for parsing log files that generates aggregated statistics and performance graphs.
  • Easy integration with the most common logging frameworks and facades: log4j, java.util.logging, Apache Commons Logging and SLF4J.
  • Custom log4j appenders to generate statistics and graphs in a running application (custom java.util.logging handlers coming soon).
  • The ability to expose performance statistics as JMX attributes, and to send notifications when statistics exceed specified thresholds.
  • A servlet for exposing performance graphs in a web application. Note: that Graphs for average execution time and transactions per second are generated as URLs to the Google Chart Server.

    Let's change our code to use Per4J configured along with log4j
    	public static List<Trade> getTrades(String userId)
    			throws TradeServiceException {
    		StopWatch stopWatch = new Log4JStopWatch();
    		stopWatch.start("TradeService.getTrades");
    		authenticate(userId);
    		authorize(userId);
    		List<Trade> trades = TradeDao.getTrades(userId);
    		stopWatch.stop("TradeService.getTrades");
    		return trades;
    
    	}
    
    As seen above the code is more clean and maintainable. The logs now will be written to log file (e.g. server.log) automatically as shown below:
    INFO  TimingLogger - start[1315017679678] time[1000] tag[getTrades_authenticate]
    INFO  TimingLogger - start[1315017680679] time[1001] tag[getTrades_authorize]
    INFO  TimingLogger - start[1315017681681] time[2000] tag[getTrades_db]
    INFO  TimingLogger - start[1315017679678] time[4003] tag[TradeService.getTrades]
    Note: The logs are self explanatory.
    

    The Performance statistics can be obtained as: java -jar perf4j-0.9.13.jar server.log

    I hope and wish some of us can adapt Per4J sooner or Later !!!

    Have a nice day !!!!
Saturday, September 3, 2011
Posted by Munish Gogna

Maintain your daily Logs with Notepad

Do you wish to keep notes where the notes keep the date automatically? Well here is an awesome trick for you. All you need is notepad.


1) Open a blank notepad file.
2) Type .LOG in all caps at the top and hit enter.
3) Now save the file.
4) After closing the file, reopen it and notice that the date & time is now listed on the second line.
5) Also notice that the cursor is ready for you to start typing on the very next line.
6) Now every time you open it, type something on the next line and then save it, when you reopen it, it will have automatically saved the date and time on the last line.

It keeps a running record of date and time for each save. Now you have a cheap diary! Congrats!

source:centralbeat
Wednesday, August 31, 2011
Posted by Munish Gogna

Popular Post

Labels

enums (1) java (2) JAX-RS (1) JPA (1) mysql (1) request 2 (1) RESTful (1) sphinx (1) tomcat (1) web service (2) ws (2)