Chip Killmar

Software Development

Chip Killmar header image 1

Installing vsftpd FTP server on Red Hat Enterprise Linux 6

December 2nd, 2013 · No Comments · Linux

The following instructions install vsftpd 2.2.2-11.el6_4.1 and create a local user account named test with FTP access.

The initial directory for this user is set to /opt/receive so that the user may upload files there.

1. Install vsftpd using yum.

# yum install vsftpd

2. Start vsftpd automatically by the OS.

# chkconfig --levels 2345 vsftpd on

3. Modify the vsftpd config file.

# vi /etc/vsftpd/vsftpd.conf

Make the following changes:

idle_session_timeout=0
data_connection_timeout=0
user_config_dir=/etc/vsftpd/vsftpd_user_conf
userlist_deny=NO

4. Add the test user and set his password.

# useradd test
# passwd test

5. Edit the allowed FTP user list. Remove all users except the test user.

# vi /etc/vsftpd/user_list

6. Set the initial FTP directory for the test user.

# mkdir /etc/vsftpd/vsftpd_user_conf
# vi /etc/vsftpd/vsftpd_user_conf/test

Add the following line to the file:

local_root=/opt/receive

7. Make /opt/receive FTP directory globally writable.

# chmod -R go+w /opt/receive

8. Start the FTP server.

# /etc/init.d/vsftpd start

9. Using an FTP client, login as the test user and transfer a test file to the receive directory.

Note: The vsftpd FTP transfer log file is located at /var/log/xferlog.

→ No CommentsTags:

MySQL Disaster Recovery

August 6th, 2012 · No Comments · MySQL

Background

You’ve accidentally deleted some or all of the rows in a table, or even dropped a schema object altogether, in a production MySQL database. You need to recover the database to the point right before the accidental statements (DML or DDL) were executed, in essence undoing them.

What You’ll Need

  • A recent backup file created using the mysqldump command;
  • The MySQL binary logs;
    • These are usually named like mysql-bin.000123.
    • The binary logs need to cover the transactions that modified the database from when the backup was taken until when the accidental SQL statements were executed.
  • The mysqlbinlog utility. This is usually bundled with MySQL installations.

How to Recover the Database

1) End all connections to the database in preparation for recovery.  As a precaution, create a backup of the broken database, and save a copy of the binary logs from the MySQL data directory.

> mysqldump --triggers --routines --no-autocommit --user=root --password --databases test > backup.sql
> mkdir ~/binlogs
> cp /mysql/mysql-bin.* ~/binlogs

2)  Drop the broken databases and restore the last known good backup.

> mysql -uroot -p -e "drop schema test;"
> cat backup.sql | mysql -uroot -p --show-warnings

3) Determine the binary log start position to use with mysqlbinlog.

a) If your mysqldump backup was executed with the option --master-data=2, find the start position in the line CHANGE MASTER TO.... For example:

-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=282582857;

In the above line the start position to use is 282582857.

b) Otherwise, create a small sample file of all the transactions that were executed around the time when the mysqldump backup was executed.

> mysqlbinlog -v --start-datetime="2014-02-14 00:00:00" --stop-datetime="2014-02-14 00:00:05" mysql-bin.000001 > start-transactions.txt

Determine the start position by identifying the log position value of the first transaction to be replayed.

# at 282582857

In the above line the start position to use is 282582857.

4) Determine the binary log stop position to use with mysqlbinlog.

Important: the stop position should exclude the accidental SQL statements!

Create a small sample file of all the transactions that were executed right before and including the accidental SQL statements.

> mysqlbinlog -v --start-datetime="2014-02-14 14:35:00" --stop-datetime="2014-02-14 14:40:00" mysql-bin.000001 > end-transactions.txt

Determine the stop position by identifying the end_log_pos value of the last transaction executed right before the transactions to be undone.

# at 282582934
#140214 14:35:50 server id 1  end_log_pos 282583020 CRC32 0x10a3dbd1    Table_map: `test`.`xxx` mapped to number 123
# at 282583020
#140214 14:35:50 server id 1  end_log_pos 282591191 CRC32 0xd03f9bd3    Delete_rows: table id 123
...
<Many more deleted rows after this.>

From the above line, the stop position is 282582934.

5) Replay the binary logs between the start and stop positions in order to recover the database.

mysqlbinlog --start-position=282582857 --stop-position=282582934 mysql-bin.000001 | mysql -uroot -p

Tip: nightly database backups can be useful because they make the mysqlbinlog replay process, and therefore restoring the database, quicker.

References

Point-in-Time (Incremental) Recovery Using the Binary Log: https://dev.mysql.com/doc/refman/5.6/en/point-in-time-recovery.html

→ No CommentsTags:

Multihomed Hosts and Java RMI

June 22nd, 2011 · No Comments · Java

It can be challenging to get Java RMI communication working right between components on multihomed hosts. To see why, it’s important to understand how RMI determines the server hostname, which ultimately ends up being passed around in RMI stubs. If a client uses an RMI stub to make a remote method call and can’t connect to the remote object’s hostname, the usual result is the following exception:

java.rmi.ConnectException: Connection refused to host: <dotted IP address>; nested exception is:
java.net.ConnectException: Connection timed out: connect
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(Unknown Source)
	at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)
	at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)
	at sun.rmi.server.UnicastRef.invoke(Unknown Source)
	at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(Unknown Source)
	at java.rmi.server.RemoteObjectInvocationHandler.invoke(Unknown Source)

Attempting to ping the host in the exception fails.

How RMI determines the server hostname is addressed in the RMI FAQ. In a nutshell, the default server hostname is the IP address returned by InetAddress.getLocalHost().getHostAddress(), e.g. 192.168.X.X. I’ve noticed that this often is the wrong IP address. For example, on my development machine this returns the IP address for my VMware NIC, which isn’t reachable by clients on my network.

Setting the RMI system property java.rmi.server.hostname to a routable hostname is the best way to fix this problem. This value can be a DNS hostname (preferred) which will be resolved by the client’s naming service, or a dotted IP address.

Another RMI system property, java.rmi.server.useLocalHostname, set to true may also work. However, if a fully qualified hostname containing a “.” character can’t be obtained from the local name service, the IP address returned by InetAddress.getLocalHost().getHostAddress() is used again as the default. Since I use NetBIOS naming on my network, hostnames don’t contain a “.” character, and the problem remains.

The workaround/evaluation section for this bug case and this post by E. McManus also contain a good explanation of what’s happening here.

→ No CommentsTags:

XPath for Namespaces

September 3rd, 2010 · No Comments · Java

Today I was confronted with the problem of obtaining a list of all namespace attributes for a given XML document.

I found that it’s possible to achieve this quickly using the XPath expression //namespace::*, which leverages the namespace axis defined in the XPath 1.0 specification.

I’m using Xalan bundled with Sun JDK 1.6, and so my code for getting a NodeList object (and then performing some post-processing on it) is as follows:

1 // ...
2 NodeList nodeList = (NodeList) xPath.evaluate("//namespace::*", document, XPathConstants.NODESET);
3 // ...

→ No CommentsTags:

Downloading JDK 1.5?

February 10th, 2010 · 1 Comment · Java

It’s more difficult to download Sun’s JDK 1.5 now (version 1.5.0_22 to be precise as I write this) that the company has officially announced its end-of-life.  Downloading it by following the normal path on Sun’s website will land you at a registration form explaining that “Sun offers Java SE for Business, a service offering that provides access to ongoing security and bug fix updates for J2SE 5.0″, presumably something you need to pay for.

However, a Google search may result in links to the JDK 1.5 download that bypass the registration form.  This link is one such example.

Note that this finding is information that’s already publicly available and I am not recommending that you use the link above.

→ 1 CommentTags:

XPath Performance

June 23rd, 2009 · No Comments · Java

XPath is usually the perfect language, providing a simple and elegant way to compute values from the content of an XML document. For example, say that you want to count the number of elements. The XPath 1.0 query for this is about as concise as it gets:

 1 XPathFactory xPathFactory = XPathFactory.newInstance();
 2 XPath xPath = xPathFactory.newXPath();
 3 long startTime = System.currentTimeMillis();
 4 Double elementCount = (Double) xPath.evaluate("count(//*)", new InputSource(new FileReader("standard.xml")), XPathConstants.NUMBER);
 5 long endTime = System.currentTimeMillis();
 6 System.out.println("Element count: " + elementCount.intValue());
 7 System.out.println("Execution time: " + (endTime - startTime) + " millis");
 8
 9 // Output
10 // =====
11 // Element count: 1666315
12 // Execution time: 36989 millis 
Code Listing 1: XPath query to count the number of elements.

 

Choose a large XML document, however, and XPath performance can suffer. For example, try executing the above expression using the 100MB example XML document available for download here. If you do using the Sun JDK 1.6, you’ll need to increase the maximum heap size of the JVM to at least 1GB. This is because Xerces, the default XPath processor included with the Sun JDK, uses a DOM parser to parse the XML document as the first step of evaluating an XPath. Based on my own tests, Xerces takes more than 35 seconds to execute the query on the 100MB XML document, and almost the entire time is spent parsing.

In this large XML document example, one much faster alternative to XPath is to implement a SAX ContentHandler.

 1 /**
 2  * @author Chip Killmar
 3  */
 4 public class CountElementsSAXHandler extends DefaultHandler {
 5     private int elementCount;
 6
 7     @Override
 8     public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {
 9         ++elementCount;
10     }
11
12     public int getElementCount() {
13         return elementCount;
14     }
15
16     public static void main(String[] args) throws Exception {
17         SAXParserFactory saxParserFactory = SAXParserFactory.newInstance();
18         SAXParser saxParser = saxParserFactory.newSAXParser();
19         CountElementsSAXHandler saxCountNamespaces = new CountElementsSAXHandler();
20         long startTime = System.currentTimeMillis();
21         saxParser.parse(new InputSource(new FileReader("standard.xml")), saxCountNamespaces);
22         long endTime = System.currentTimeMillis();
23         System.out.println("Element count: " + saxCountNamespaces.getElementCount());
24         System.out.println("Execution time: " + (endTime - startTime) + " millis");
25
26         // Output
27         // =====
28         // Element count: 1666315
29         // Execution time: 7229 millis
30     }
31 }
Code Listing 2: SAX ContentHandler to count the number of elements.

 

As you can see, the SAX implementation requires more effort than executing an XPath, but not much. The element counter above executes against the 100MB XML document for me in just over 7 seconds – that’s more than 5 times faster than XPath.

→ No CommentsTags:

Pretty-Print XML from a DOM

March 25th, 2009 · 4 Comments · Java

If you’re like me it can be useful to convert a DOM in Java into a formatted XML string. Pretty-printing a DOM Element can augment the information gleaned from your debugger when tracking down a  bug, for example.

I’ll show you three ways to pretty-print XML from a Java DOM below.  All of them format a Document but can be easily modified to serialize any Node.  (Note: if you’re interested, you can download source code for this posting here.)

Method 1: TrAX

The JAXP Transformation API for XML is a logical first choice for pretty-printing XML.

 1 static String prettyPrintWithTrAX(Document document) throws TransformerException {
 2     // Pretty-prints a DOM document to XML using TrAX.
 3     // Note that a stylesheet is needed to make formatting reliable.
 4     TransformerFactory transformerFactory = TransformerFactory.newInstance();
 5     Transformer transformer = transformerFactory.newTransformer(new StreamSource("pretty-print.xsl"));
 6     StringWriter stringWriter = new StringWriter();
 7     StreamResult streamResult = new StreamResult(stringWriter);
 8     DOMSource domSource = new DOMSource(document);
 9     transformer.transform(domSource, streamResult);
10     return stringWriter.toString();
11 }
Code Listing 1: Using TrAX to pretty-print XML.

 

Using the TrAX API is straightforward (Code Listing 1), however dealing with whitespace in a DOM is rather tricky.  It’s important to note that if your DOM was parsed from XML containing whitespace, this will by default be preserved in the object model.  Furthermore, the default TrAX indentation engine (Xalan-Java) won’t reformat existing whitespace. 

Therefore, to reliably pretty-print XML with TrAX you’ll need to supply an XSLT stylesheet (Code Listing 2) that removes whitespace from the DOM during transformation.  This is a subtle point about serialization using TrAX that’s missed on the Xalan website which can conflate a bug found in JDK 1.5.  You can read more about this phenomenon here and here

 1 <?xml version="1.0" encoding="UTF-8"?>
 2 <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xalan="http://xml.apache.org/xslt" version="1.0">
 3     <xsl:output method="xml" encoding="UTF-8" indent="yes" xalan:indent-amount="4"/>
 4     <!--Important!  Remove existing whitespace in DOM elements.-->
 5     <xsl:strip-space elements="*"/>
 6     <!--Identity transformation (see http://www.w3.org/TR/xslt#copying).-->
 7     <xsl:template match="@*|node()">
 8         <xsl:copy>
 9             <xsl:apply-templates select="@*|node()"/>
10         </xsl:copy>
11     </xsl:template>
12 </xsl:stylesheet>      
Code Listing 2: An XSLT stylesheet for pretty-printing.

 

Method 2: DOM Level 3 Load and Save

DOM Level 3 Load and Save (LS) is the new API for pretty-printing XML.  Support for LS is included with JAXP 1.3 which is bundled as part of Sun’s JDK 1.5, but unfortunately pretty-print formatting only works in JDK 1.6 or better.

The code to use LS’s LSSerializer is more involved:

 1 static String prettyPrintWithDOM3LS(Document document) {
 2     // Pretty-prints a DOM document to XML using DOM Load and Save's LSSerializer.
 3     // Note that the "format-pretty-print" DOM configuration parameter can only be set in JDK 1.6+.
 4     DOMImplementation domImplementation = document.getImplementation();
 5     if (domImplementation.hasFeature("LS", "3.0") && domImplementation.hasFeature("Core", "2.0")) {
 6         DOMImplementationLS domImplementationLS = (DOMImplementationLS) domImplementation.getFeature("LS", "3.0");
 7         LSSerializer lsSerializer = domImplementationLS.createLSSerializer();
 8         DOMConfiguration domConfiguration = lsSerializer.getDomConfig();
 9         if (domConfiguration.canSetParameter("format-pretty-print", Boolean.TRUE)) {
10             lsSerializer.getDomConfig().setParameter("format-pretty-print", Boolean.TRUE);
11             LSOutput lsOutput = domImplementationLS.createLSOutput();
12             lsOutput.setEncoding("UTF-8");
13             StringWriter stringWriter = new StringWriter();
14             lsOutput.setCharacterStream(stringWriter);
15             lsSerializer.write(document, lsOutput);
16             return stringWriter.toString();
17         } else {
18             throw new RuntimeException("DOMConfiguration 'format-pretty-print' parameter isn't settable.");
19         }
20     } else {
21         throw new RuntimeException("DOM 3.0 LS and/or DOM 2.0 Core not supported.");
22     }
23 }
Code Listing 3: Pretty-printing XML with DOM Level 3 Load and Save.

 

Method 3: XMLSerializer

If you’re still stuck using JDK 1.5, don’t worry. You can configure Xerces-J‘s XMLSerialzer directly to pretty-print XML.  Although this is the easiest method, it depends on an internal class in the Xerces parser and is not JAXP compliant!  You can read more in Question 11 of the JAXP FAQ.

 1 static String prettyPrintWithXMLSerializer(Document document) throws IOException {
 2     // All is not lost if you're still on JDK 1.5: just use XMLSerializer with the appropriate OutputFormat.
 3     // The following will pretty-print the DOM document to XML.
 4     StringWriter stringWriter = new StringWriter();
 5     XMLSerializer serializer = new XMLSerializer(stringWriter, new OutputFormat(Method.XML, "UTF-8", true));
 6     serializer.serialize(document);
 7     return stringWriter.toString();
 8 }
Code Listing 4: Pretty-printing XML using XMLSerializer.

 

→ 4 CommentsTags:

Rebuilding Thumbnails in Gallery 2

February 4th, 2009 · 9 Comments · Miscellaneous

Have you ever wanted to recreate thumbnails and resized picture copies for your photos in Gallery 2?

I did recently after making a settings change to the ImageMagick image processing library and was surprised that it wasn’t a more straightforward process.

[Read more →]

→ 9 CommentsTags:

Something You Didn’t Know About Generics

August 4th, 2008 · No Comments · Java

Before the introduction of Generics in Java 5 it’s commonly known that in a class you can’t have two methods with the same signature that differ by return type alone.  More formally, the Java Language Specification §8.4.2 says that “it is a compile time error to declare two methods with override-equivalent signatures in a class”.  Take for example the following code which does exactly this and unsurprisingly doesn’t compile:

 1 // Generates error "myMethod(java.util.List) is already defined
 2 // in DoesNotCompile when compiled with Sun JDK version 1.5.0_13.
 3 abstract class DoesNotCompile {
 4     abstract String myMethod(List listOfStrings);
 5
 6     abstract Integer myMethod(List listOfIntegers);
 7 }

Code Listing 1

It might be surprising though that a class with the same methods changed slightly to use Generics compiles without errors:

 1 abstract class Compiles {
 2     abstract String myMethod(List<String> listOfStrings);
 3
 4     abstract Integer myMethod(List<Integer> listOfIntegers);
 5 }

Code Listing 2

[Read more →]

→ No CommentsTags:

Evaluating Excel Formulas with Apache POI

April 28th, 2008 · No Comments · Java

Apache POI is a full-featured Java library for reading several different Microsoft file formats, including Excel. (Note: this post only covers POI version 3.0.2-FINAL released in February 2008).

POI supports evaluating cells with formulas containing Excel functions. For example, consider the code below for reading a java.util.Date from a simple spreadsheet with the formula “=NOW()” in cell A1 which outputs Mon Apr 28 22:48:57 CDT 2008:

 1 HSSFWorkbook workbook = new HSSFWorkbook(inputStream);
 2 HSSFSheet sheet = workbook.getSheet("Sheet1");
 3 HSSFRow row = sheet.getRow(0); // 0-based.
 4 HSSFCell cell = row.getCell((short) 0); // Also 0-based;
 5 System.out.println(cell.getDateCellValue()); 
 

Code Listing 1: Reads cached formula result.

This works great! But on closer examination, you’ll notice that the value returned is the saved result of the last formula evaluation in the spreadsheet, meaning that it will always return the same (in this case) Mon Apr 28... value. What should you do if you want a dynamic value from the formula, e.g. when the program is run tomorrow?

[Read more →]

→ No CommentsTags: